Merwinsson Posted November 29, 2017 Share Posted November 29, 2017 I am rebuilding my existing drive pool to a new one and I would like to do the following: I have 8 disks in the new pool. Call them D1 thru D8. I would like to have new files added to D1, then D2 when D1 reaches 95% capacity, then to D3, then to D4, always honoring the 95% capacity rule. As for the other disks, D5 will be the duplicates location for D1, and likewise D6 for D2, D7 for D3, and D8 for D4. Can I achieve this? If so, how? Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted November 29, 2017 Share Posted November 29, 2017 Yup! Check out the "Ordered File Placement" balancer plugin. https://stablebit.com/DrivePool/Plugins Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted December 14, 2017 Author Share Posted December 14, 2017 Well, I'm a bit confused about that PlugIn, and I've read the notes, so maybe you can help me out here. I have all plugins installed (although most are disabled), and my pool has pool duplication turned on. See attached picture. Right now my pool is balanced and I am happy about that. Now I want to move to the scheme I mentioned. It is not clear to me how to get the duplicate files to go on the disks I want. To recap... I want any file created on D1 to have its duplicate created on D5, and likewise for D2->D6, D3->D7, and D4->D8. I guess I can enable "Ordered File Placement" and set D1 thru D4 as the disks with the highest priority. However, I don't want new files going on D5 and higher, ever, and there seems to be no way to "say" that. Likewise there doesn't seem to be any plugin/config that enables me to ensure exactly how to place a duplicate of a file. Maybe you can just tell me what to enable and what to configure, and I'll do it. I don't want to play with a 48TB pool and make a mistake. Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted December 15, 2017 Share Posted December 15, 2017 Unfortunately, there isn't a good way to do this, because of how the balancing system is designed. Additionally, unless there is a specific reason for this, there is no actual need to do this, either. Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted December 15, 2017 Author Share Posted December 15, 2017 There is a *desire* to do this because I don't want to go hunting thru 8 disks to find a file should the pool break. If I have originals on D1-D4 and dups on D5-D8 then I only need search through D1-D4. Also, the pool is gonna get 2 more disks not too much later in the future, so that's more disks to sift though. I didn't think this scheme is a big deal but can it be done by relaxing the exact location of the dups, as in, still put the dups on D5-8 but which disk it ends up on is not like my original req where the dup file has to end up on a specific disk according to where its original was, but rather it ends up *somewhere* on D5-D8. Can I do that? Quote Link to comment Share on other sites More sharing options...
0 Umfriend Posted December 15, 2017 Share Posted December 15, 2017 Couldn't you use hierachical Pools for this? 1. Set-up Pool 1 (say U:\) with HDDs D1 to D4, no duplication, file placement as desired 2. Set-up Pool 2 (say V:\) with HDDs D5 to D8, no duplication, file placement as desired 3. Set up Pool 3 (say W:\) with U:\ and V:\, x2 duplicattion. Done! Having said that, what do you mean by "go hunting thru 8 disks to find a file should the pool break"? If it is about one HDD failing then ypu do not need to sift thure disks as DP would re-duplicate the files that were on that HDD for you, for free and automativally (once you have removed the failing HDD from the Pool and provided you have sufficient space). And for that scenario, 1 Pool of 8 HDDs is better even. Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted December 15, 2017 Author Share Posted December 15, 2017 While I didn't realize you can create a pool using other pools, I don't see how this helps. In Umfriend's example above pool W will behave exactly like a x2 pool with all 8 disks - a new file can end up anywhere on D1 thru D8, and its dup likewise. I want new files to go to D1 thru D4 ONLY, and dups to D5 thru D8 ONLY. In my original example I was requiring that a dup end up on a very specific disk based on where the original was placed, but I'm relaxing that and saying dups can go anywhere on D5 thru D8, but they should never go on D1 thru D4. It would be nice to have more definite and configurable duplication rules or a plugin to access them. For example, the Drive Usage Limiter plugin UI is close, but no cigar. My whole pool is duplicated, nothing is unduplicated, so this doesn't help. Or does it? What if I did this...? In my pool disks D5 thru D8 are "B" type disks (my disks are actually labeled 1A,1B,2A,2B, etc). Maybe the above would prevent new files from appearing on D5 thru D8...but I think it would also prevent the dups of those files from going to D5 thru D8, and that would be bad; also, some dups would still end up on D1 thru D4. What I think we need is THIS: Catch my drift? StableBit peoples...can we get something like this, pleeeeease. Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted December 15, 2017 Share Posted December 15, 2017 The problem is that there is no "duplicate" file. Both are (should be) 100% identical. Neither is an "original" or "copy". Both are "the same". Adding the concept of original and duplicate has a bunch of issues. And that's not including the fact that it would take a significant rewrite of the balancing and duplication code, and some additional overhead on the pooled drives ... and modifications of file system commands.... Unless there is a reason for doing this, it's not worth the effort. However, what you could do (and if I wasn't tired *and* sick when I posted yesterday, I would have thought of this)... Pool D1 and D5, enable duplication Pool D2 and D6, enable duplication Pool D3 and D7, enable duplication Pool D4 and D8, enable duplication Then pool these pools together and DO NOT ENABLE DUPLICATION on the top level pool. That would get you what you want, just maybe not as cleanly as you'd like. Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted December 15, 2017 Author Share Posted December 15, 2017 I was afraid you were going to say there was no distinction between either copy of a file that is duplicated. Oh well. I see how your suggested scheme would help me but how do I go from the scheme I have to your scheme without extra temporary storage to backup the existing pool before tearing it all down and starting over? Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted December 15, 2017 Share Posted December 15, 2017 Well, it looks like you can remove a disk or two right now. That may be the simplest. The other option is to use the "dpcmd"s "ignore-poolpart" option to break the pool. Then add the disks to the pool, as above, and then "double seed" the pool. The "ignore-poolpart" command wants the "xxxxx" from the "PoolPart.xxxxx" folder on the pool. So it would look like "dpcmd ignore-poolpart s: xxxxxx". And then you repeat this for each disk. You will get missing disk notifications, but ignore this for now. And you can find PoolPart folders by checking each disk, with the "Show Hidden Files" option enabled. The, for the "double seeding", it's basically, this: http://wiki.covecube.com/StableBit_DrivePool_Q4142489 But in each PoolPart folder, there will be a second PoolPart folder. Move everything into that, instead. Then reset the settings. You may end up with stuff that isn't duplicated properly, or too much stuff for a specific disk. If so, then move stuff around, as needed. Quote Link to comment Share on other sites More sharing options...
0 Umfriend Posted December 16, 2017 Share Posted December 16, 2017 6 hours ago, Merwinsson said: While I didn't realize you can create a pool using other pools, I don't see how this helps. In Umfriend's example above pool W will behave exactly like a x2 pool with all 8 disks - a new file can end up anywhere on D1 thru D8, and its dup likewise. I want new files to go to D1 thru D4 ONLY, and dups to D5 thru D8 ONLY. In my original example I was requiring that a dup end up on a very specific disk based on where the original was placed, but I'm relaxing that and saying dups can go anywhere on D5 thru D8, but they should never go on D1 thru D4. It would be nice to have more definite and configurable duplication rules or a plugin to access them. For example, the Drive Usage Limiter plugin UI is close, but no cigar. Actually, no. One copy would be on D1 to D4, the other on D5 to D8. This is because Pool W will regard Pool U and Pool V as single HDDs and never place both duplicates on only U or ony V but rather one on U and one on V. If you then use the optional Ordered File Placement plug-in, you would fill D1 and D5 first, then D2 and D6 and so on (this does assume that D1 is od equal size as D5, D2 as D6 and so on). Some organise otherwise with the File Placement Rules. I wouldn't know but it may require that you offload data first indeed. But the question remains, why bother? Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted December 16, 2017 Author Share Posted December 16, 2017 It seems my planned scheme could be done, but would be a major headache getting there. I'd have to bring 2 disks out of the pool, wait for that, re-pool them, add them back, re-balance, wait for that, then repeat this process for 2 more disks, until all 8 are done. Eh, forget it. I'll just live with my vanilla pool for now. Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted December 17, 2017 Share Posted December 17, 2017 unfortunately, yeah, a pretty big headache. Quote Link to comment Share on other sites More sharing options...
0 Merwinsson Posted January 23, 2018 Author Share Posted January 23, 2018 I'd like to second the idea of adding a balancer option to spread duplicate files evenly over controllers as asked in this thread: I'd rather not create a pool of pools. This really shouldn't be too hard. Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted January 23, 2018 Share Posted January 23, 2018 8 minutes ago, Merwinsson said: I'd like to second the idea of adding a balancer option to spread duplicate files evenly over controllers as asked in this thread: I'd rather not create a pool of pools. This really shouldn't be too hard. Conceptually, it's not a hard concept. The implementation, on the other hand. To allow this, AND to keep the existing balancer and file placement options, "complex" is an understatement. For the foreseeable future, the only way to do this is with Hierarchical pooling. Quote Link to comment Share on other sites More sharing options...
Question
Merwinsson
I am rebuilding my existing drive pool to a new one and I would like to do the following:
I have 8 disks in the new pool. Call them D1 thru D8.
I would like to have new files added to D1, then D2 when D1 reaches 95% capacity, then to D3, then to D4, always honoring the 95% capacity rule.
As for the other disks, D5 will be the duplicates location for D1, and likewise D6 for D2, D7 for D3, and D8 for D4.
Can I achieve this? If so, how?
Link to comment
Share on other sites
14 answers to this question
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.