Drivepool not balancing

[Wendo]

I've had Drivepool for a while now and I've never managed to get it to balance successfully. I came from Drivebender where this just worked without issue.

 

I've reset settings completely and it still won't do it. The logs report this for every balance attempt ever. Even manually starting a balance and clicking on "Increase priority"

 

9:37:49.5: Information: 0 : [FileMover] Bucket lists scanned all pool parts. (FillRatio=0.00)
9:37:49.6: Information: 0 : [FsControl] Set overall pool mode: PoolModeNormal (lastKey=Rebalance, pool=cefc11e5-7b22-4038-9b7d-60dc0a598925)
9:37:53.6: Information: 0 : [Rebalance] Setting Background I/O priority
9:37:53.6: Information: 0 : [Rebalance] Cannot balance. Immediate re-balance throttled.

 

This is on a 5 drive pool with zero other activity so I've no idea why it thinks it's being throttled. The OS is a VM on ESX with disks passed through is that matters. How can I force it to ignore any throttling and just do a damn balance run?

 

Thanks

[Wendo]

Ok, so the throttling appears to be because the default of don't balance more than once every 12 hours was on. Without that I get his from the logs. I'm trying to balance by percentage used. Disk Space Equalizer is the #1 balancer in the list but it never balancers. 300GB free on one 100TB drive, 2.5TB free on the other and a 700GB difference of free space over the other three 4TB disks

 

10:37:44.8: Information: 0 : [Rebalance] Balancing pool parts...
10:37:44.8: Information: 0 : [FileMover] Building bucket lists for 1 pool parts:
10:37:44.9: Information: 0 : [FileMover]   scanning pool part '\\?\Volume{ec93b34b-2a9d-472d-a4d3-209521fd19fa}\PoolPart.65378e06-f717-4143-b36c-afa943a6ddd0\' (FillRatio=0.00)...
10:43:50.4: Information: 0 : [FileMover] Bucket lists scanned all pool parts. (FillRatio=0.00)
10:43:50.5: Information: 0 : [FsControl] Set overall pool mode: PoolModeNormal (lastKey=Rebalance, pool=cefc11e5-7b22-4038-9b7d-60dc0a598925)
10:43:50.6: Information: 0 : [Rebalance]   calculating drive fill ratio (UIDs: 65378e06-f717-4143-b36c-afa943a6ddd0)...
10:43:50.6: Information: 0 : [Rebalance]     (Limit=9,893,320,847,360 B, UsedSpace=9,666,062,909,440 
10:43:50.6: Information: 0 : [Rebalance]     not over limit (1.0)
10:43:50.6: Information: 0 : [Rebalance]   calculating drive fill ratio (UIDs: af6a9d1b-4ba5-426c-bbb4-a4d6111c7993)...
10:43:50.6: Information: 0 : [Rebalance]     (Limit=10,395,075,227,123 B, UsedSpace=7,615,991,062,003 
10:43:50.6: Information: 0 : [Rebalance]     not over limit (1.0)
10:43:50.6: Information: 0 : [Rebalance]   calculating drive fill ratio (UIDs: c4afa782-b5d8-47b7-a635-497290f92065)...
10:43:50.6: Information: 0 : [Rebalance]     (Limit=3,893,276,700,672 B, UsedSpace=2,734,561,628,160 
10:43:50.6: Information: 0 : [Rebalance]     not over limit (1.0)
10:43:50.6: Information: 0 : [Rebalance]   calculating drive fill ratio (UIDs: 024074b2-551d-4af0-9811-defc71f2ac60)...
10:43:50.6: Information: 0 : [Rebalance]     (Limit=3,893,276,700,672 B, UsedSpace=2,141,979,611,136 
10:43:50.6: Information: 0 : [Rebalance]     not over limit (1.0)
10:43:50.6: Information: 0 : [Rebalance]   calculating drive fill ratio (UIDs: a39b059e-2675-4755-9710-40a6e205a986)...
10:43:50.6: Information: 0 : [Rebalance]     (Limit=3,893,276,700,672 B, UsedSpace=2,063,574,085,632 
10:43:50.6: Information: 0 : [Rebalance]     not over limit (1.0)
10:43:50.6: Information: 0 : [Rebalance]   Calculating optimal slack...
10:43:50.6: Information: 0 : [Rebalance]     Largest PoolSize (UIDs: af6a9d1b-4ba5-426c-bbb4-a4d6111c7993, Largest=10,502,449,409,523 
10:43:50.6: Information: 0 : [Rebalance]     Other PoolSize (UIDs: 65378e06-f717-4143-b36c-afa943a6ddd0, Other=9,999,285,065,761 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=503,164,343,762)
10:43:50.6: Information: 0 : [Rebalance]     Other PoolSize (UIDs: 024074b2-551d-4af0-9811-defc71f2ac60, Other=4,000,320,542,456 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]     Other PoolSize (UIDs: a39b059e-2675-4755-9710-40a6e205a986, Other=3,999,309,184,672 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]     Other PoolSize (UIDs: c4afa782-b5d8-47b7-a635-497290f92065, Other=3,999,304,066,077 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]   Optimal slack: 0 B
10:43:50.6: Information: 0 : [Rebalance]   Calculating duplication obstruction...
10:43:50.6: Information: 0 : [Rebalance]   Calculating current slack...
10:43:50.6: Information: 0 : [Rebalance]     Largest FreeSpace (UIDs: af6a9d1b-4ba5-426c-bbb4-a4d6111c7993, Largest=2,886,458,347,520 
10:43:50.6: Information: 0 : [Rebalance]     Other FreeSpace (UIDs: a39b059e-2675-4755-9710-40a6e205a986, Other=1,937,076,797,440 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=949,381,550,080)
10:43:50.6: Information: 0 : [Rebalance]     Other FreeSpace (UIDs: 024074b2-551d-4af0-9811-defc71f2ac60, Other=1,858,671,271,936 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]     Other FreeSpace (UIDs: c4afa782-b5d8-47b7-a635-497290f92065, Other=1,266,089,254,912 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]     Other FreeSpace (UIDs: 65378e06-f717-4143-b36c-afa943a6ddd0, Other=334,632,120,320 
10:43:50.6: Information: 0 : [Rebalance]       (Largest=0)
10:43:50.6: Information: 0 : [Rebalance]   Current slack: 0 B
10:43:50.6: Information: 0 : [Rebalance]   Obstructed: 0 B
10:43:50.6: Information: 0 : [Rebalance]   Other pooled files: 16,606,178,234,368 B
10:43:50.6: Information: 0 : [Rebalance]   (DuplicationObstructionSlack=0 

 

[Shane]

To maximise the likelihood of balancing occurring, ensure you have "Automatic balancing - Triggers" set to 100% with "Or at least this much data needs to be moved" ticked and set to a small value (e.g. 100 MB).

[Wendo]

No change sadly, with the file mover logging turned up you can see it build a list of 50000+ files it thinks need to move, and then it just doesn't so it knows it needs to balance, it just doesn't

[Shane]

Try temporarily turning off all other balancers and requesting a re-measure?

[Wendo]

Sadly just the same outcome

[Shane]

@Christopher (Drashna) Any ideas?

[Christopher (Drashna)]

Honestly, not enough information.  Missing stuff like the balancing ratio reported on the pool, etc. 

That said, setting the balancing ratio to 100% should make it balance very aggressively (when/where it needs to).  But the default balancing settings shouldn't move the data around except for in specific edge cases.  Such as the disks being over 90% full,  there being "unsuable for duplication" space reported on the pool, etc. 

That said, could you open a ticket at https://stablebit.com/contact and after doing so, run the StableBit Troubleshooter, using the contact ID number from the ticket? 

 

[Wendo]

The bit of info I apparently failed to add, was that I'm trying to balance to Equalize by percent used and my disks look like this. Even if it isn't supposed to use more than 90% DRIVE2 is more than that and never being cleaned out. I'll open a ticket when I get a bit more spare time and see if we can get to the bottom of this

 

[Wendo]

For anyone that finds this later. The system is now balancing for free space on each drive. 

 

At this point the only thing I can assume was that because one drive was over 90% full it just completely broke balancing with the Drive Space Equalizer plugin, and that failure cascaded and broke balancing for all plugins. I had the Prevent Drive Overfill set to not go over 90%, but I'm wondering if any plugin will actually bring a drive under 90% usage (if that's what it's set too) once it's already gone over. All the text seems to imply it will just stop it going over 90%, nothing about recovering from such a situation.

 

I disabled all plugins except the disk usage limiter and told it not to put any unduplicated data on DRIVE2. That ran successfully and got DRIVE2 under 90%, and after that and enabling the other plugins it started working exactly as I'd expected it to do so for the last year.