Jump to content
Covecube Inc.
  • 0

An idea to moving files simultaneously form SSD to multiple HDDs with balancing


Tarh

Question

I know this has been discussed a few times and I more or less understand the issues and limitations, but I was wondering if it would be possible to give the balancer a bit of a leeway with the free space based balancing logic.

What I mean is by default the balancer is putting files on the disk with the most free space. Great. If the next file in the queue would go to another drive (because the actively copied file makes the first target disk less free than another disk) we now have two files being copied and so on. Sounds fast and great, except this is very rare so the SSD is almost always only feeding and is limited by one HDD - which we're trying to avoid as it sucks, even without SMR drives. Like an NVMe SSD cache writing at like 100MB/s is painful. The drive fills faster than it offloads.

So, what if we could give the balancer an optional threshold to be more flexible or in another words less strict with only pushing files to the least filled drive? Surely then it could push files to multiple drives at once in a default setup.

For this theoritical threshold, let's say if we used the value 5%, then basically the balancer would start copying to the least filled drive as it does now, then it would look at the rest of the drives as if they had 5% more free space (respecting fill limits) therefore it could use them to push files simultanously. Obviously only copying one file to one HDD at a time (better performance, less fragments, as it is now). With a theoritical secondary option we could limit how many threads it could do at once, or just a built in one thread per one HDD (which is how it already seem to work). This way, given enough SSD bandwidth, the SSD could feed each drives at once until it's empty most of the time instead of when very rarely the stars align.

 

I don't believe this would be overly complex (like multithreading) and I think it would make many people's lives better who use SSD caches. Unless I miss some major issue.

Link to post
Share on other sites

1 answer to this question

Recommended Posts

  • 0
9 hours ago, Tarh said:

The [SSD] drive fills faster than it offloads.

Having the SSD cache flush out to 2 or more HDDs at the same time sounds like a good improvement. I wish I had that problem of filling the SSD drive faster than it offloads. My download computer transfers files to my DrivePool server over my home network at ~30 MB/s (limitation of my download computer components) but my SSD cache in DrivePool flushes off to any HDD in the pool at ~60 MB/s. So, I never have seen my SSD cache overfill on my setup. But I can see the advantages of multi-threads flushing the SSD cache on a faster system than mine.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...