Drivepool does not distribute files across the array in real time?

[RockmanXX]

Hi guys,

I am using drivepool + snapraid setup (with 1 x 8TB, 5 x 4TB) and try to achieve a scenario that "all new files distribute across the storage array in real time while the location of old files in the array remain unchanged whien balancing", so that all new files i copied to the array will distribute accordingly and snapraid won't heavy load when sync (since the balancing caused problems as they move around the files (old and new ones).

I was wondering how can i setup correctly to achieve that?

below is what i tried:

• Default setting which forcing my new files to only the largest disk first (Data 2 8TB). (Please ignore the other DATA disk as i was testing the balancing configuration)  

 

• Disk space Equalizer which i have setup with "balance based on percentage" and it distribute the files in a intended way but ONLY AFTER the transfer completed (transfer all files to DATA 2) and perform re-balance manually. This causing problems for snapraid after re-balance as the plug-in moving files around.

 

• Ordered file placement, again, I thought it is the order of "distributing the files" like file 1 -> Data2, file 2 -> Data 1, file 3 -> data 3 etc.) but no, it filled up all new files to Data 2 first, when the limit (rule) is reached, i think it should THEN move on to Data 1.  

 

Correct me if i am wrong, it does not distribute all the new files in real time? Below is my balancing setup:

 

Anyone can guide me the correct way to achieve the goal? Thanks!~

[Shane]

Hi! The short answer is that the goal of "all new files distribute across the storage array in real time while the location of old files in the array remain unchanged whien balancing" can't generally be accomplished because DrivePool considers "old files" to be "all files already in the pool" while SnapRaid considers "old files" to be "any file that is neither new nor changed since the last sync".

Before continuing, it may be useful to understand the following about DrivePool:

• Placement of existing files on the pool, a.k.a. balancing, is controlled by the Balancers and the File Placement rules.
• Placement of new files on the pool, a.k.a. real-time file placement, defaults to whichever drive(s) have the most free space at the time.
  • This is done primarily to minimise the risk of having insufficient free space to complete the writing of a new file.
  • The real-time file placement limits that can be set, by the File Placement rules and by some Balancers, can override this.
• If you hover your mouse cursor over each of the Balancers the tool-tip will tell you if it can set real-time file placement limits.
• Unfortunately the Drive Space Equalizer balancer is not one of those able to set a real-time file placement limit.

So it's generally recommended that DrivePool's balancers be turned off (except maybe Scanner) when used together with SnapRaid, to avoid the problems it causes with syncing. DrivePool will still distribute new files across a pool's disks according to their free space (or under whatever placement limits you've set) and that still results in an even distribution of used space across those disks over time if they are of the same size.

In your case however you have one data disk that is significantly larger than the rest, and that skews the new file distribution accordingly.

I'd suggest turning off the balancers and either:

• Accept the skew; once the free space remaining on the 8TB data disks falls to the same level as the 4TB data disks then additional new files will tend to spread evenly over time, there'll just always be 4TB extra data on the disk that is 4TB bigger than the others. The simplest alternative.
• Shrink the partition on the 8TB disk to 4TB for now or replace the 8TB disk with a 4TB disk; new files will spread evenly without the skew. The next simplest alternative.

I hope this helps.

[RockmanXX]

Appreciate your response and if it works in these way, i will deal with it.

But trying to understand a bit more before this post end, in my first image it is clearly that the 8TB is significantly more than other 4TB.

So for example:

now my 8TB contain 4.45TB data and my other 4TB contain 0.82TB data

so drivepool will work correctly when the 8TB reach 4.82 TB if I turn off all the balancers?

Or which balancer you recommended to use it for minimal balancing while using snapraid?

Or any recommendation for settings i should follow (such as do not balance automatically) to achieve similar to my goal while using snapraid with it?

For your Point 2, if i did not get it wrong, i can cut out 2 4TB partitions from the 8TB and add them all to the pool so that i should get what I want right XD?

Thanks!~

[Shane]

Basically with all balancers/placement off, DrivePool prefers to fill whatever disk has the most free space at the time of accepting a file.

"now my 8TB contain 4.45TB data and my other 4TB contain 0.82TB data so drivepool will work correctly when the 8TB reach 4.82 TB if I turn off all the balancers?"

Yes, after that point DrivePool will tend to alternate between them as each gets less free space than the other from receiving new files.

"Or which balancer you recommended to use it for minimal balancing while using snapraid?"

With Snapraid, and considering no other factors? Only the Scanner balancer, if at all. I feel the extra load during a sync from dealing with an evacuated drive is preferable to having to rebuild a drive from parity, but that's just my feeling and your use case and mileage may vary.

"For your Point 2, if i did not get it wrong, i can cut out 2 4TB partitions from the 8TB and add them all to the pool so that i should get what I want right XD?"

If you're just wanting to get all the bar graphs in DrivePool to line up prettily, yep that's a way. Just note you'd then need to use a minimum of two parity levels of protection instead of one in your SnapRaid config (I'm assuming you currently have a single 8TB parity disk or multiple split disks totalling 8TB of single parity somewhere since you have an 8TB data disk) because if your 8TB disk partitioned as 2x4TB partitions failed you'd simultaneously lose two "disks" from SnapRaid's perspective.

If however you were wanting to maximise that free space on the 8TB to keep in reserve for some reason... maybe you could do something like use the SSD Optimizer balancer and time/script it so that your regular SnapRaid syncs happen just after the 8TB disk finishes emptying the new files into all the 4TB disks. And/or put in a feature request for the Drive Space Equalizer to be able to set real-time file placement limits and hope that's feasible.

DrivePool and SnapRaid can work very well together... so long as you are careful to line them up nicely.

[RockmanXX]

thanks for recommendation, have a nice day!!~😄