Jump to content
Covecube Inc.
  • 1
monty1977

Not balancing onto new drives

Question

Hello!

Just moved away from my Synology NAS after a bad experience trying to get files off it when the hardware failed - total pita!  

I've got the following drives in a pool with folder duplication set on most folders:

8 Tb
2 x 4Tb
2 x 3Tb

..12.8Tb currently free, duplication free, nice green bar.  It's not balancing any of the files to one of the 3Tb and one of the 4Tb drives though.  Is this normal?  Is it just not bothering as there's so much free space elsewhere?

Thanks!


 

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

@Christopher (Drashna),

I just started evaluating DrivePool and see there is still not an option for real-time balancing for file transfers based on percent used. 

I would like this option to help optimize disk read performance since it would help equalize files being written to the outer space of the hard drive platters.

I understand the potential issue you noted in your previous post. But, is it possible to get around this scenario by identifying the size of the file being transferred and store it on a disk with sufficient space if the drive with the lowest percent used doesn't have capacity?

Share this post


Link to post
Share on other sites
  • 0
7 hours ago, AlanO said:

@Christopher (Drashna),

I just started evaluating DrivePool and see there is still not an option for real-time balancing for file transfers based on percent used. 

I'm confused - if you have real time balancing turned on, then this option has existed since... forever? Just make sure the disk space equalizer is above all other plugins besides Scanner (if you use that).

image.png.4bf7a28022240495a8adcd066fad8181.png

Share this post


Link to post
Share on other sites
  • 0
On 9/7/2018 at 2:16 PM, Christopher (Drashna) said:

StableBit DrivePool's default file placement strategy is to place new files on the disks with the most available free space.  This means the 1TB drives, first, and then once they're full enough, on the 500GB drive.    So, yes, this is normal.

The Drive Space Equalizer doesn't change this, but just causes it to rebalance "after the fact" so that it's equal.  

@Chris Downs, Disk Space Equalizer is the only balancer I'm using and it is above all others in the list. Based on the comment quoted above, my understanding is that new files will still be placed on the drive with the most free space and Disk Space Equalizer would still then move the files to another drive after that. So, while the end result may be the similar, it's not the same as having the new file initially placed on the drive with the least % used.

Having this balancing done after the fact results in an extra data transfer, once to the pool(to the drive with most free space) and then to the drive with the least % used. This extra step can take considerable time when double digit terabytes are involved. So, my inquiry was really more about being able to have new files initially placed on the drive with the least % used, provided the disk has sufficient space of course, and not having the need for balancing after the fact.

To be honest, I have not enabled the option to balance immediately as I was under the impression this wouldn't help as it would still have to balance after new data is written to the pool.

Share this post


Link to post
Share on other sites
  • 0
12 hours ago, AlanO said:

@Chris Downs, Disk Space Equalizer is the only balancer I'm using and it is above all others in the list. Based on the comment quoted above, my understanding is that new files will still be placed on the drive with the most free space and Disk Space Equalizer would still then move the files to another drive after that. So, while the end result may be the similar, it's not the same as having the new file initially placed on the drive with the least % used.

Having this balancing done after the fact results in an extra data transfer, once to the pool(to the drive with most free space) and then to the drive with the least % used. This extra step can take considerable time when double digit terabytes are involved. So, my inquiry was really more about being able to have new files initially placed on the drive with the least % used, provided the disk has sufficient space of course, and not having the need for balancing after the fact.

To be honest, I have not enabled the option to balance immediately as I was under the impression this wouldn't help as it would still have to balance after new data is written to the pool.

I see, so then you need the opposite option in the balancer - "Equalize by free space remaining", I think? This should then give you what you want, for incoming new files?

Share this post


Link to post
Share on other sites
  • 0

@Chris Downs,

No, that wouldn't seem to work either. That strategy would still rely on the balancer to do it's job AFTER the data has been transferred to the pool. Again, according to the comment I quoted above, DrivePool will always place new files on the disk with the most free space and then rebalance the data between the drives according to the selected balancers. 

So, basically forget about balancers as they act AFTER the data has already been written to the pool. What I am desiring is an option to initially place new files on the drive with the least % used at the time the file is written.

Share this post


Link to post
Share on other sites
  • 0

Disclaimer: Not a DrivePool dev.

@AlanO I suspect the problem with "identifying the size of the file being transferred" is that DrivePool presents itself as a standard NTFS-formatted single-volume drive to the Windows file system interface, which (I presume) means it has to deal with the fact that Windows expects to be able to create files of undefined length whenever it wants.

TLDR while Windows may first ask "how much free space do you have", when it goes "I'm going to start writing a file, I'll tell you when I'm done" then the only choice a NTFS drive - physical or virtual - has is to go "yes boss, I'll tell you if I run into any problems". The drive does not get to ask questions.

Share this post


Link to post
Share on other sites
  • 0

Not a dev either.

Another problem might be that once one starts writing a file, it may not be yet known how large that file will be. Sure, a simple copy should not have that issue but what if you save a stream of as yet unknown length? Writing to the disk with most space free seems to be the safest option (and personally, I like safe). Also, I find it hard to think of a scenario where a subsequent reshuffle of files would measurably impact performance.

Share this post


Link to post
Share on other sites
  • 0

Hi all, i'm new to drivepool (switching from windows storage spaces). Sorry if my english is not perfect ;)

 

I will try to explain a little bit my config and use case. I have a pool with 5 disks :

  • 1x 8 To
  • 2 x 6 To
  • 1x 4 To
  • 1 x 500 Go (special disk for my use case, with non pool folder)

All disks are in the same bay, plug in USB. I don't care of file placement, so i have activated only the disk space analyzer balancer and let him do the job. and it works like a charm.

 

I have some folders with a lot of small files (2ko), and some others with a lot of files between 1.5/2Go. The grow is about 2K/day for small files, and 20 to 30 for big files.

Files are kept and the delete is only manual, when i do not need the file for more work.

 

So when a file is added, the real time behavior of Drivepool is to send them to the 8 To. And when placement is too bad, the balancer trigger is rised and move files to others disks. But as it is a single USB interface, it's very easy to overload with R/W for move all files.

I'm very interested to know if there is a way (or a plan to add that) to configure different strategy for realtime. Is it something plugin can do?

 

Thanks in advance

Share this post


Link to post
Share on other sites
  • 0

I'm kind of following, I think... what would your preferred strategy be? (for example,  "I want new files to go to X, then Y, then Z.... and I want it to balance old files at 1am to Y, Z and J so X is freed up again.")

Share this post


Link to post
Share on other sites
  • 0

The behavior of storage space is not so bad on this point. When you add files, it fill disk by step of 10%. 

What can be an idea is to be able to configure the step (1%, 5%, etc...). 

 

Currently i'm adding files to my pool, they are added to "J:". Balancer will be rised after that, and move them on G, I or F. H is not allowed on file placement to receive this folder with a lot of day usage, so balancer will only move other files on it. I think in a couple of days/weeks, balancer will end to move other files on H and is usage will grow with my "legacy" files.

image.png.38679c4c8b714fefc663b82009a14d9e.png

What i expect in my dream, when i add a file, the pool will set the file on the disk with the less % of disk usage (if disk free space is enough, and if file placement rules allow it), and evaluate this for each file. If I add 10 files of 2Go, the pool will dispatch them on all disks in real time.

Then the balancer can be rise after (every day, or if files placement is too bad) to reequilibrate if required because the real time can sometime result of not optimal placement.

My goal is to equilibrate the load for write and after for read by balancing it on multiple disks, instead if for now i'm limited with the single USB interface, it's enough to work on 2 or 3 disks are the same time. And in background, i work to optimize that to add more interface for disks in the pool.

 

I checked the plugin wiki but it seems that plugin can only balancer file after, not be able to be invoked for real time. Another good idea can be to add a way for us to develop our real time plugin. I'm a c# developer and very interested if this can be done to give it a try to create it ;) 

Share this post


Link to post
Share on other sites
  • 0

It should be possible to write a balancer that controls real time file placement, since that's what the SSD Optimizer and the Ordered File Placement balancer plugins do.

DP defaults to writing to the disk with the most free space in bytes rather than by percentage to minimize the possibility of running out of space on the destination disk during writing, because it is not possible to guarantee the size of the write in advance (for example, a 1 TB GB disk with 10% free space has only a fifth of the space available of a 10 TB disk with 5% free space).

Share this post


Link to post
Share on other sites
  • 0

Not a bad idea, seems plugin can set a limit on real-time placement

image.thumb.png.7684c7365d308462b8cf40583c24fd6a.png

 

So an idea can be to set a limit on all disks except the one we want to use, and recompute the limitation regulary. Legacy balancer will correct placement in background if necessary.

I will try and let you know

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...