Jump to content
Covecube Inc.
  • 1
monty1977

Not balancing onto new drives

Question

Hello!

Just moved away from my Synology NAS after a bad experience trying to get files off it when the hardware failed - total pita!  

I've got the following drives in a pool with folder duplication set on most folders:

8 Tb
2 x 4Tb
2 x 3Tb

..12.8Tb currently free, duplication free, nice green bar.  It's not balancing any of the files to one of the 3Tb and one of the 4Tb drives though.  Is this normal?  Is it just not bothering as there's so much free space elsewhere?

Thanks!


 

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

Were all of the drives empty and added to the Pool before files were placed on it?  Perhaps some of the files were manually copied into the hidden Poolpart-xxxxx folders in the root of each?  Hard to tell without a screenshot and knowing how you populated the pool.  :) 

What you can do to even things out, is install the Disk Space Equalizer plugin for Drivepool and force a manual re-balance.  You go into "Manage Pool", then Balancing, then enable it in the Balancers tab.  Hit Save after that, and Drivepool will kick off a manual re-balance.  When it's done balancing, toggle that plugin off again.  If you have any doubts about whether or not the pool display is correct, just force a re-measure before starting the manual re-balance.

Share this post


Link to post
Share on other sites
  • 0
On 9/3/2018 at 6:24 PM, Jaga said:

Were all of the drives empty and added to the Pool before files were placed on it?  Perhaps some of the files were manually copied into the hidden Poolpart-xxxxx folders in the root of each?  Hard to tell without a screenshot and knowing how you populated the pool.  :) 

What you can do to even things out, is install the Disk Space Equalizer plugin for Drivepool and force a manual re-balance.  You go into "Manage Pool", then Balancing, then enable it in the Balancers tab.  Hit Save after that, and Drivepool will kick off a manual re-balance.  When it's done balancing, toggle that plugin off again.  If you have any doubts about whether or not the pool display is correct, just force a re-measure before starting the manual re-balance.

Awesome, seems to be working now.  FYI all drives were freshly formatted/bitlockered drives with nothing on them when I added them to the pool

Thanks!

Share this post


Link to post
Share on other sites
  • 0
On 9/3/2018 at 7:57 AM, monty1977 said:

..12.8Tb currently free, duplication free, nice green bar.  It's not balancing any of the files to one of the 3Tb and one of the 4Tb drives though.  Is this normal?  Is it just not bothering as there's so much free space elsewhere?

Basically, yes.

The software's default placement strategy is to place new files on the drive with the most available free space, measured absolutely.
That means that the 8TB would be used first, then the 4TB drives, and then the 3TB drives, in that order.

Share this post


Link to post
Share on other sites
  • 0

Hello! I will post here instead of creating a new topic as I experience similar issue.

I use Disk Space Equalizer plugin with "Equalize by percent used" setting. I had 5x 1TB pool, and it was distributing new files equally onto those 5 HDDs on the fly.

 

However, after adding one new 500GB HDD and manually balancing the pool, all new files are still placed onto those five 1TB hdd and nothing goes into 500GB HDD so I have to balance it manually. No settings were changed and it stopped equalizing files properly after adding that new 500GB HDD.

One workaround is to use automatic balancing with "If the balance ratio falls below 100%" but its not elegant solution - it keeps files moving around the pool almost continuously instead of placing them right away onto the right HDD.

2018-09-06_183511.jpg

2018-09-06_183526.jpg

It simply looks like drives added after creating pool are not taken under consideration by balancing plugin. 

Now I added another empty 320GB and its the same behavior. No files copied into Pool are written onto those 500 & 320GB HDDs, only 5 1TB drives that were used to create original pool. 

Share this post


Link to post
Share on other sites
  • 0
1 hour ago, cichy45 said:

It simply looks like drives added after creating pool are not taken under consideration by balancing plugin. 

Now I added another empty 320GB and its the same behavior. No files copied into Pool are written onto those 500 & 320GB HDDs, only 5 1TB drives that were used to create original pool. 

You seem to be using the Drive Usage Limiter plugin - have you checked it's settings to make sure the 500 and 320 drives aren't excluded from receiving either duplicated or unduplicated content?  And do you have any regular file placement rules set in the File Placement tab?  Either could impact those newer drives' ability to receive files.

It might also be helpful to see a snapshot of your pool's main interface, so we can see the drives on the right and their positional arrows.  It lets us know what plans Drivepool has for filling each.

Share this post


Link to post
Share on other sites
  • 0

@Jaga

 

I do not have any file placement rules as I don't mind on which HDD files are placed. When I disable all plugins except Disk Space Equalizer nothing changes. I made small experiment with small partitions. I started another pool with 2 partitions (G & F), filled them with some data, all was balanced +/- OK. Then I added another small partition ( i ) and it was not "noticed" by equalizer plugin neither before or after manually balancing. Manually balancing is working with Disk Space Equalizer - however it is not able to work correctly in real time with added disks/partitions after filling pool with some data.

Screen of Test Pool:

 

2018-09-06_205024.jpg

Here is screen with my main Pool (320GB removed already): 

2018-09-06_205116.jpg

 

Balancing plugin seems to "know" (considering that "balance to" arrow) to fill 500GB HDD as its is lacking behind, but it is not doing so.

Share this post


Link to post
Share on other sites
  • 0
6 hours ago, cichy45 said:

Balancing plugin seems to "know" (considering that "balance to" arrow) to fill 500GB HDD as its is lacking behind, but it is not doing so.

So when you add a new file to that pool, does it land on the I: drive?  You might also want to force a re-measure, so that the pool can check all files/volumes and determine for itself if it's doing the right thing.

Share this post


Link to post
Share on other sites
  • 0

@Jaga

No, it doesent and that is the problem. I can use manual balancing, to equalize all (old and new disks), but if I start copying new data onto Pool, these new drives wont recieve any of it. All data will be distributed only between old disks. 

So, to make it clear:

1. I create Pool with 5 drives.

2. I copy some data to pool, Disk Space Equalizer Plugin is equalizing placing of data between all 5 disks in real time.

3. I add new disk to this old Pool.

4. I start to copy new data again, but DSEP is still placing it only on 5 disks that were used to create Pool.

5. I do Manual Balancing proces, now 5 old and 1 new disk is balanced correctly.

6. DSEP is still not putting any data on new disk when I copy something into pool after balancing manually.

 

 

Interesting fact. Even when I disable all plugins, DrivePool will still place data only on original 5 HDDs. Remeasuring pool doesn't change anything. Only manual balancing seems to be able to put new data onto new disk. 

 

It looks like DSEP/DrivePool is able to distribute data between old disks in real time, but in will fill new disks only after old are full.

Share this post


Link to post
Share on other sites
  • 0
39 minutes ago, cichy45 said:

Interesting fact. Even when I disable all plugins, DrivePool will still place data only on original 5 HDDs. Remeasuring pool doesn't change anything. Only manual balancing seems to be able to put new data onto new disk. 

Now that IS weird, since by design it's supposed to be auto-balancing all new additions (subject to file/drive placement rules of course, if they exist).  I haven't had this issue happen before - I even added two new 8TB data drives to my 7x drive pool recently, forced a manual re-balance, and it's been correctly balancing new additions ever since.

Barring any suggestions from @Christopher (Drashna) or other community members, the only thing I could suggest is doing a full reset on Drivepool's settings.  When it's done it'll restart the service, and re-measure / re-check consistency.  At that point, change the settings back to what you had, and see how it behaves.

Share this post


Link to post
Share on other sites
  • 0

@Jaga

 

Tried that settings-reset but didn't work. As you can see, (copying from D: to DrivePool) it will only put data on those five 1TB disks. The only difference is that 500GB HDD is attached via SAS Perc H310 flashed to IT mode, but that shouldn't make any difference.

(Don't mint that HDS723020BLA642 MN6OA5R0 as it's a snapraid parity drive for Pool so it is only used when syncing).

 

I will wait, maybe one of developers/administrators will be able to provide more information on this error I encounter. I really like DrivePool and it would be perfect for my needs, but lack of this feature (real-time disk equalization for added disks) is almost a deal-breaker. It would be cool if they could make it a core functionality (like the default ordered file placement) instead of being an plug-in that could go outdated and incompatible after some DrivePool release.

2018-09-06_234632.jpg

Share this post


Link to post
Share on other sites
  • 0

Well, with the "Balance Immediately" setting enabled, AND the Disk Space Equalizer plugin enabled, you get on-the-fly real time balancing across all pool members.  That applies to not only new files added, but existing ones already in the pool.  It's hard on the drives though, since a lot of additions/changes/removals does a lot of shuffling around on the drives.  But perhaps the combination is what you're looking for.

Typical behavior for the pool is to put new files onto the drives with the most "absolute" space free.  That may be why you're seeing files hit the larger drives instead of the smaller one - they have more space overall.  But I'd think the Disk Space Equalizer with percentage checked would override that.

Share this post


Link to post
Share on other sites
  • 0

Yeah, I think that's what Disk Space Equalizer was made for - to equalize space between HDDs with different capacity. I suspect that this plugin might be a bit outdated and thus don't work as expected. 

Share this post


Link to post
Share on other sites
  • 0

The plugins do have to match the version of the program they were designed for.  It's worth checking to make sure you have the latest version of both.

Share this post


Link to post
Share on other sites
  • 0

Maybe I am just simply misunderstanding this plugin and its is not able to equalize disks in real time? Only after being triggered (manually or automatically). If that is the case, I would be a bit disappointed. 

 

I hope @Christopher (Drashna) will be able to confirm or deny this ability of Disk Space Equalizer plugin. If it can not do it in real time, maybe it would be a nice feature to consider? 

 

Basically, I would like DrivePool to fill all my Pooled disks to the same % of space used. If new disks are added, all new data copied into Pool will go onto these new HDD until they are the same % used as the rest of old HDD in pool. All on-the-fly while new files are written into Pool, instead of manually/auto triggering balancing. 

I have a lot of HDD that are various capacity, from 80 to 2000GB so this behavior would be great feature for me as I would be able to add new disks without triggering balancing, new files would be used to balance the pool instead of moving files from old HDD to these newly added.

 

3 disks, 2TB, 2TB and 1TB used in 10%, so they hold 200GB, 200GB and 100GB respectively. I add one new 3TB, and all new data is written only into that new 3TB until it is used in 10% (holding 300GB of data). 

After being filled to 300GB, all new files should be distributed on-the-fly between these 4 HDD so they will have constant X % of space used.

Share this post


Link to post
Share on other sites
  • 1
23 hours ago, cichy45 said:

4. I start to copy new data again, but DSEP is still placing it only on 5 disks that were used to create Pool.

To clarify a couple of things here (sorry, I did skim here):

StableBit DrivePool's default file placement strategy is to place new files on the disks with the most available free space.  This means the 1TB drives, first, and then once they're full enough, on the 500GB drive.    So, yes, this is normal.

The Drive Space Equalizer doesn't change this, but just causes it to rebalance "after the fact" so that it's equal.  

So, once the 1TB drives get to be about 470GB free/used, it should then start using the 500GB drive as well. 

 

There are a couple of balancers that do change this behavior, but you'll see "real time placement limiters" on the disks, when this happens (red arrows, specifically).  If you don't see that, then it defaults to the "normal" behavior. 

Share this post


Link to post
Share on other sites
  • 0
1 hour ago, Christopher (Drashna) said:

The Drive Space Equalizer doesn't change this, but just causes it to rebalance "after the fact" so that it's equal.  

That's good to know, thanks for that info Christopher.  Always wondered if it applied live, or retroactively.

Share this post


Link to post
Share on other sites
  • 0

Thanks for your answer @Christopher (Drashna)

Would it be possible for developers, to create a new file placing strategy, to replicate behavior of Drive Space Equalizer but in real time? So users could switch between "place on HDD with most space" or "use equal % of space on every HDD".

Share this post


Link to post
Share on other sites
  • 0

@Umfriend

 

I have a random collection of hard drives, from a 80 to 2000GB. I would like to put them all into one pool. One reason is that SnapRAID is faster with more drives, as it is calculating parity from all physical HDD at once. Second, if data is distributed between more disks, if something goes wrong - less of it will be lost.

Third - I just "feel better" (yes, that is totally personal preference) when all of hdd in pool are used instead of spinning for nothing. 

Default file placement is great if all of your HDDs are the same size. It is great for my 5x 1TB pool, but it starts to get messy when I add 2TB, 320GB, 500GB, 1.5TB etc., drives.

 

Yes, I could just Pool two 500GB to make one 1TB Pool, and pool that pool into pool to get equal % of disk space used (10% of 1TB pool = 10% out of each 500GB) but... you can see where it is going with even more random disk like 750, 640GB etc. That is why I would like to have that option build into DrivePool.

Share this post


Link to post
Share on other sites
  • 0

I see. With 80 vs 2000, I guess I understand although, really, ditch the 80 :D

I don't know snapraid. Does it do parity filebased or blockbased, i.e., all HDDs entirely? But with 500 and 2000, why bother? If you use the Pool enough, the 500 will get used. Anyway, you know you situation best, was just wondering.

Share this post


Link to post
Share on other sites
  • 0

Snapraid does parity filebased, so you can defrag your HDD, delete/add files and it will calculate parity only for these changes. You can also force it to recalculate parity for all files if you want.

For me, on 5x 1TB pool it's going 300-500MB/s, not too fast as each drive can do between 120-180MB/s but my NTFS is compressed so its additional CPU load (+ I got a lot of small files, so HDD can't hit their max transfer speed). Some people hit 1500MB/s when calculating parity and 2500MB/s when checking hash on pools with multiple HDDs. 

Share this post


Link to post
Share on other sites
  • 0

It could, but one of the issues that you'd run into .... that 80GB drive is 50% full, and you try to add a 50GB file to the pool..... that 40GB free on that drive may cause the free space check to outright fail, and give you a "Not enough free space" error. 

 

That's why the default file placement strategy is handled this way.  To avoid this sort of issue. 

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...