I know this has been discussed a few times and I more or less understand the issues and limitations, but I was wondering if it would be possible to give the balancer a bit of a leeway with the free space based balancing logic.
What I mean is by default the balancer is putting files on the disk with the most free space. Great. If the next file in the queue would go to another drive (because the actively copied file makes the first target disk less free than another disk) we now have two files being copied and so on. Sounds fast and great, except this is very rare so the SSD is almost always only feeding and is limited by one HDD - which we're trying to avoid as it sucks, even without SMR drives. Like an NVMe SSD cache writing at like 100MB/s is painful. The drive fills faster than it offloads.
So, what if we could give the balancer an optional threshold to be more flexible or in another words less strict with only pushing files to the least filled drive? Surely then it could push files to multiple drives at once in a default setup.
For this theoritical threshold, let's say if we used the value 5%, then basically the balancer would start copying to the least filled drive as it does now, then it would look at the rest of the drives as if they had 5% more free space (respecting fill limits) therefore it could use them to push files simultanously. Obviously only copying one file to one HDD at a time (better performance, less fragments, as it is now). With a theoritical secondary option we could limit how many threads it could do at once, or just a built in one thread per one HDD (which is how it already seem to work). This way, given enough SSD bandwidth, the SSD could feed each drives at once until it's empty most of the time instead of when very rarely the stars align.
I don't believe this would be overly complex (like multithreading) and I think it would make many people's lives better who use SSD caches. Unless I miss some major issue.
Question
Tarh
I know this has been discussed a few times and I more or less understand the issues and limitations, but I was wondering if it would be possible to give the balancer a bit of a leeway with the free space based balancing logic.
What I mean is by default the balancer is putting files on the disk with the most free space. Great. If the next file in the queue would go to another drive (because the actively copied file makes the first target disk less free than another disk) we now have two files being copied and so on. Sounds fast and great, except this is very rare so the SSD is almost always only feeding and is limited by one HDD - which we're trying to avoid as it sucks, even without SMR drives. Like an NVMe SSD cache writing at like 100MB/s is painful. The drive fills faster than it offloads.
So, what if we could give the balancer an optional threshold to be more flexible or in another words less strict with only pushing files to the least filled drive? Surely then it could push files to multiple drives at once in a default setup.
For this theoritical threshold, let's say if we used the value 5%, then basically the balancer would start copying to the least filled drive as it does now, then it would look at the rest of the drives as if they had 5% more free space (respecting fill limits) therefore it could use them to push files simultanously. Obviously only copying one file to one HDD at a time (better performance, less fragments, as it is now). With a theoritical secondary option we could limit how many threads it could do at once, or just a built in one thread per one HDD (which is how it already seem to work). This way, given enough SSD bandwidth, the SSD could feed each drives at once until it's empty most of the time instead of when very rarely the stars align.
I don't believe this would be overly complex (like multithreading) and I think it would make many people's lives better who use SSD caches. Unless I miss some major issue.
Link to comment
Share on other sites
1 answer to this question
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.