Lycan Posted July 27, 2018 Share Posted July 27, 2018 Brand new to Drivepool, looks like lots of features and options, myriad of possibilities, a bit overwhelming to a new user. Running an 8 x 500GB Samsung 860 with 2 x Seagate 8TB HDDs. What I want to do is have all source files on the SSDs, and then 2 replicas of each source file on each of the 8TB HDDs. Yes, it is probably overkill having 5 total copies of each file, but that is what I'm shooting for. How would I best define this? Plugins? Balancer settings? Thanks! Quote Link to comment Share on other sites More sharing options...
0 Jaga Posted July 27, 2018 Share Posted July 27, 2018 Hierarchical child pools would probably serve you best. But you'll need to partition each 8TB drive into 2 logical 4TB volumes each. To my knowledge it isn't possible to run duplication (either pool or folder) on a single drive listed in a pool. Pool 1: All SSDs are member disks in this pool. No pool duplication. Pool 2: A 4TB volume from HDD1 + a 4TB volume from HDD2. Enable 2x pool duplication, Real Time Duplication, and Read Striping. Pool 3: The other 4TB volume from HDD1 + the other 4TB volume from HDD2. Enable 2x pool duplication, Real Time Duplication, and Read Striping. Main Pool: Made up by adding in Pools 1/2/3. Enable 3x pool duplication, Real Time Duplication, and Read Striping. Then add whatever drive letter you want to the Main Pool and start dumping files into it. It'll distribute a copy to each of the three child pools (1 2 and 3). Each HDD's first 4TB volume will get a copy and then duplicate it, and Pool 1's SSDs should distribute the files equally among them. If you run Anti-Virus on that machine, also enable Bypass File System Filters on all pools to enhance performance. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 27, 2018 Author Share Posted July 27, 2018 TYVM! Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 27, 2018 Author Share Posted July 27, 2018 In the process of breaking the pool into the pieces as suggested, first step was to disable duplication. However, even with all duplication disabled, I'm still showing about 300GB worth of duplicated files. What steps should be taken to clear out the duplication stragglers? Quote Link to comment Share on other sites More sharing options...
0 Jaga Posted July 27, 2018 Share Posted July 27, 2018 There's always going to be -some- duplication on a Pool with multiple disks. Usually that's the Metadata, and the larger the structure/files you have, the larger the Metadata. If you do properties on your folder structure you can get an idea how many folders/files total are on the pool. Drivepool tries to duplicate the info on these (Metadata) three times, if there are three available drives in the pool. If not, it'll try for two times duplication on them. However if you want to re-calc the Pool's duplication, click on the little gear in the upper right, choose troubleshooting, then "recheck duplication...". If Drivepool finds any invalid dupes, it should clean them up itself. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 27, 2018 Author Share Posted July 27, 2018 Ahh, the old re-measure seems to have cleared things up. I was wondering why it kept showing my SSDs as being larger than they were. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 27, 2018 Author Share Posted July 27, 2018 I modified the lines Pool 2: A 4TB volume from HDD1 + a 4TB volume from HDD2. Enable 2x pool duplication, Real Time Duplication, and Read Striping. Pool 3: The other 4TB volume from HDD1 + the other 4TB volume from HDD2. Enable 2x pool duplication, Real Time Duplication, and Read Striping. Reasoning that with that layout, one drive failure takes out both. Split each in half, both halves of each drive pooled together. D1P1 + D1P2 = Pool2 D2P1 + D2P2 = Pool3 Quote Link to comment Share on other sites More sharing options...
0 Umfriend Posted July 27, 2018 Share Posted July 27, 2018 Your reasoning is sound but I wonder whether DP will then duplicate for Pool2 and Pool3 as Duplication requires different *physical* HDDs. In fact, if D1 or D2 fails, you lose both duplicates on that HDD so why have two copies on one HDD anyway? Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 27, 2018 Author Share Posted July 27, 2018 You are correct, it would not allow it. So sticking with 3x duplication, 1 x for each sub-pool. Thanks again for your time. Quote Link to comment Share on other sites More sharing options...
0 Jaga Posted July 27, 2018 Share Posted July 27, 2018 Yep - the only way however to get 4 copies on two drives was to divide it logically. You did after all, request 5 total copies, 4 of which were on 2 physical disks. You still have multiple HDD copies even in the event of one 8TB HDD failure - the other HDD would have two copies on it's two volumes, so file redundancy would continue, just on the single working HDD. It could protect against things like file system corruption on it's 2nd volume, leaving you with a single good copy on the first volume of the non-failed drive. It's very redundant and normally you wouldn't partition the drive that way, but your Pool scheme required it to equally spread things out and work well. 34 minutes ago, Umfriend said: In fact, if D1 or D2 fails, you lose both duplicates on that HDD so why have two copies on one HDD anyway? Two reasons I structured it that way for him. 1) That was the only way I saw to get >2 copies on just 2 drives with duplication working. 2) You still get redundancy in the event one volume is unusable due to physical sector damage, or file system corruption, full drive failure, etc. Imagine a scenario with this setup where one of the HDDs holding Pool 2 fails completely. It leaves child Pools 2 and 3 degraded, each with two copies of the pool contents on it. Then imagine that one of the volumes on Pool 2 is also damaged but at the file level (or even damaged disk sectors affecting a volume). A file he needs isn't readable. We *still* have the remaining volume on Pool 3 with another copy. If the physical disk is still good, and the file system isn't corrupt/damaged, it's retrievable over there and still serves as a backup for the SSD Pool 1. Highly redundant I agree, but that's what was asked for. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 29, 2018 Author Share Posted July 29, 2018 Indeed I did Now if only the performance of the SSD side of the pool didn't tank to the performance level of the HDD archive disks. Tried with read-striping enabled and disabled, still seems to want to read from HDDs as much as the SSDs. Quote Link to comment Share on other sites More sharing options...
0 Jaga Posted July 29, 2018 Share Posted July 29, 2018 That is good info for Christopher and Alex. In the case of child pools, it may be a nice future feature to add in some type of priority selection, or prediction, such that faster sub-pools/drives are read from first. But it sounds like you have the structure all setup and working, which is great. Quote Link to comment Share on other sites More sharing options...
0 Umfriend Posted July 29, 2018 Share Posted July 29, 2018 On 7/27/2018 at 6:12 PM, Jaga said: Two reasons I structured it that way for him. 1) That was the only way I saw to get >2 copies on just 2 drives with duplication working. 2) You still get redundancy in the event one volume is unusable due to physical sector damage, or file system corruption, full drive failure, etc. Oh, I fully agree with your solution to the OP's requirement, just questioned the requirement. The requirement also slows write performance as a single file needs to be written twice to the same HDD and perhaps this even increases the likelihood of a drive failure (not sure but I would guess that writes do add to wear & tear). Three duplicates is rather secure already, on top of that I would sooner look into actual backup solutions (incl. offsite/rotation). Edit: I never paid much attention to read striping but when I did I never thought it worked well. @OP: You've got at least 11 HDD/SSD's attached, how/what MB? I would not be suprised if the 8xSDDs were attached through some sort of card/device that affected read-striping. Quote Link to comment Share on other sites More sharing options...
0 Jaga Posted July 29, 2018 Share Posted July 29, 2018 2 minutes ago, Umfriend said: ...on top of that I would sooner look into actual backup solutions (incl. offsite/rotation). Well, I'll agree with you there, considering the slowdown the (archive/shingled?) HDDs introduce. They're among the slowest ones for writes you can get today I think. I had no idea they were shingled drives. It would probably be easier to use a single Pool (consisting of just the SSDs) and perform a backup to one of the 8TB HDDs, then mirror that to the other HDD with something like file sync software. It would give triple redundancy and keep the pool as fast as possible. It would even serve to protect against accidental file deletions, as well as using differentials/incrementals to give a recoverable file history. Using backup compression would keep writes to a minimum on those archive drives, essentially increasing throughput and extending lifespan. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 29, 2018 Author Share Posted July 29, 2018 I probably will end up breaking them up and using something like rsync / or the old Microsoft SyncToy to synchronize directories. I also have an old LG nas with dual mirrored 6TB Drives. Some of my kit is upwards of 5 to 6 years old now, the 8 port SAS/SATA controlller is amongst the oldest. Intel Core i7-4790k @ 4.00Ghz, 32GB Corsair Extreme Profile 1, Gigabyte GA-Z97X-Gaming 7 Motherboard, LSI SAS2 2008 Falcon in JBOD mode, 2 x 4 Pair Breakout cables, tied to IcyDock 8 Bay SSD Hot Swap, 8 x Samsung 850/860 500GB SSDS, 1 x Samsung 960 Evo NvME, 2 x 8TB Seagate HDDs, 1x2TB Seagate HDD in 3 Bay Hot Swap Enclosure, AMD Radeon R9 390 8GB Video Card, Samsung CHG9_C49HG9xDM (3840 x 1080 in one 49" monitor) Quote Link to comment Share on other sites More sharing options...
0 Umfriend Posted July 29, 2018 Share Posted July 29, 2018 That's a nice collection of hardware! The 8TBs, Archives or regular HDDs? Connected to the main board ports? I am not sure how DP determines the drive to read from, I think it is the "bus speed" which may not actually be the HDD speed. Chris would know. Quote Link to comment Share on other sites More sharing options...
0 Lycan Posted July 29, 2018 Author Share Posted July 29, 2018 The 3 regular HDDs are tied to the mainboard sata controllers, along with the the BluRay burner, the case's built in drop-in 3.5/2.5 HDD/SSD socket and 1 x eSata port and NvME drive. The SAS2 controller is dedicated to the 8 x SSDs. The case is a Thermaltake Chaser Mk-1, lots of room for cooling, liquid and air, room for internal enclosures, etc... The SAS2 2008 Falcon started life as a 9211-8i before the firmware update/conversion to JBOD only, with BIOS disabled. I use it as my Gaming Rig, VM Server Rig, Media Storage Rig. Quote Link to comment Share on other sites More sharing options...
Question
Lycan
Brand new to Drivepool, looks like lots of features and options, myriad of possibilities, a bit overwhelming to a new user.
Running an 8 x 500GB Samsung 860 with 2 x Seagate 8TB HDDs.
What I want to do is have all source files on the SSDs, and then 2 replicas of each source file on each of the 8TB HDDs.
Yes, it is probably overkill having 5 total copies of each file, but that is what I'm shooting for.
How would I best define this? Plugins? Balancer settings?
Thanks!
Link to comment
Share on other sites
16 answers to this question
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.