Jump to content
  • 0

Randomize parallel file writes to different drives


phhowe17

Question

I am using a file sync tool that can run multiple file copies at one time.  The problem I'm having is that DrivePool is picking the same physical disk for all four file write streams.  I have 10gb network cards between a server and it's backup and with parallel file copies to a single drive limits performance.

The pool is generally will balanced before the copies start.  Using an SSD cache is not really a solution as often the total volume would exceed 500GB.

Is there a way to force DrivePool to spread file operations across multiple disks?

Link to comment
Share on other sites

4 answers to this question

Recommended Posts

  • 0

DrivePool's programmed behaviour is to write to the poolpart that has the most free space at the time each write operation begins because write operations can be open-ended, and it wants to minimise the chance of running out of space on any given poolpart (in the typical usage scenarios). You could make a feature request via the Contact form to have an advanced option to override this. For example, "I'd like an advanced option to have the pool select drives for writing based on minimum utilisation instead of maximum space". Edit: or perhaps round-robin between the first N poolparts with the highest free space. A problem is that the more checks DP has to do when picking which poolpart to use, the more "seek time" each write operation will have.

Alternatively, if your tool supports doing so and you understand the risks involved (e.g. same-but-different file conflicts), you could write each stream directly to a different poolpart folder; e.g. use the SSD balancer to set four drives as "SSD" (even if they're HDD) such that their free space is maximised over the others and then have the tool copy one set of files to poolpart A while copying another set of files to poolpart B etc, and then manually remeasuring the pool afteerwards.

Link to comment
Share on other sites

  • 0
13 hours ago, Umfriend said:

Assuming there is no duplication, wouldn't a single 2TB SSD suffice as a cache? Way simpler for a somewhat specific use case IMHO.

Perhaps, I'm not sure that wouldn't be limited by my bus, I don't have an M.2 slot in this system.  I think I'll try some small SATA ssds and see if I have enough to do RAID 5 and have Drivepool use them as cache.  Otherwise I'll take Shane's suggestion and split up some of my existing HD into 500GB + remainder and mark the 500GB segments as "SSD"

Most of the drive are on a LSI 9xxx-8i so are sharing that one controller.  Maybe adding another controller would help.

Link to comment
Share on other sites

  • 0

In RAID might work. Just adding a few drives as SSD Cache may run into the same problem I fear. Also, one SSD, even if it is SATA, will outperform 4 HDDs I think. I am assuming you don't have actual SAS-drives. Depending on the actual chipset, two RAIDed SSDs might do the job fully.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...