Balance to multiple drives when receiving multiple incoming file transfers

[blondvillain]

I'm in the midst of transferring a large volume of data from an 8-bay NAS into a new DrivePool setup and I'm trying to speed up the process a bit. I tried using the SSD plugin with a 500 GB SSD to enable greater bandwidth for multiple file copy operation simultaneously, but the balancer can't keep up with the workload and the SSD fills up faster than it can evacuate. The issue seems to be that while the balancer is evacuating files from the SSD, it only places them on one drive at a time, despite the fact that several of the drives in the pool are below the "full" threshold. Many of the drives in my pool are slow Seagate Archive drives that have poor write speeds, so if the balancer limits itself to writing to only one drive at a time, it'll never be able to keep up with large volumes of incoming data.

 

How can I configure DrivePool to balance incoming transfers across multiple drives in the pool simultaneously so that it can better handle large volumes of incoming data, either with OR without an SSD landing zone? The Archive drives are able to sustain reads at the normal 100+ MB/s, but writes are extremely variable, dropping into the single digits at times. Therefore, my pool is presently unable to accomodate the full speed of even ONE single incoming transfer. If it could be configured to balance incoming transfers across multiple drives simultaneously, it would really speed up the process.

 

I've also been evaluating Drive Bender, and while I prefer DrivePool's interface, Drive Bender appears able to send multiple incoming transfers to separate drives in the pool without issue, and without any special configuration required.

 

Just FYI, I'm NOT doing any file duplication or redundancy.

 

[Spider99]

i have several Seagate Archive drives and have no slow writes especially when doing sequential write which would be expected with copying large amounts of data to a pool - mine work at between 100MB/s up to 220MB/s - i.e. inside to outside of the disk respectively

 

your ssd should be able to copy to multiple hdd as well as receive data across your lan - mine do and i have duplication on and when copying to the pool across a 10G network

 

maybe you have other issues which is slowing things down?

[blondvillain]

i have several Seagate Archive drives and have no slow writes especially when doing sequential write which would be expected with copying large amounts of data to a pool - mine work at between 100MB/s up to 220MB/s - i.e. inside to outside of the disk respectively

 

your ssd should be able to copy to multiple hdd as well as receive data across your lan - mine do and i have duplication on and when copying to the pool across a 10G network

 

maybe you have other issues which is slowing things down?

 

The fact that Drive Pool only writes to one drive in the pool at a time is slowing things down quite substantially. How is yours configured to allow balancing to multiple drives at the same time? This is what I'm trying to accomplish, but the various configurations I've tried have failed to accomplish this.

 

Also, it is well established that Seagate Archive drives highly variable sustained write performance. They use Shingled Magnetic Recording (SMR) technology, as opposed to Perpendicular Recording (PMR), and variable writes is a characteristic of SMR by design, and the main reason why Seagate Archive drives are so much cheaper than PMR alternatives of the same capacity. A quick Google search for reviews will turn up a lot of evidence supporting this. For instance http://www.storagereview.com/seagate_archive_hdd_review_8tb. Here's a quote from said review.

 

"Below is a screenshot showing disk activity during the SMR RAID rebuild on top, where we see sustained write performance all over the map, including single digit throughput for long periods. This is compared to the PMR rebuild shown on the bottom half of the image which is able to stay over 100MB/s for most of the duration."

 

So I don't know what drives you're talking about, but I have the ones that have variable write performance.

 

Returning to the topic at hand: could you please share your Drive Pool configuration settings that enable balancing to multiple drives simultaneously?

[Christopher (Drashna)]

The copying/writing to one drive at a time is normal. 

 

Windows' file copy is single threaded, meaning that it only copies one file at a time. And for the most part, this is a good thing. the more you're doing in parallel, the slower it will be overall.  So single threaded is faster, overall. 

 

As for the balancing, ... well, the balancing uses the normal file transfer APIs, and yes, it's done single threaded.  In most cases, multithreading the balancing, so that multiple copies wouldn't be helpful and could adversely affect performance ... as well as add unnecessary complexity to the balancing (what files to copy from where to where).

 

 

Also, You've posted this in a ticket, so I'll post the response here: 

 

 

Unfortunately, I'm not sure there is a way to really speed this up, at least in a way that will change things significantly. 

 

Specifically, the way that the balancer is designed is that files are written to the "SSD" drives first, and then balanced off the drive.  Unfortunately, this balancing is a slower process.    Both because we run the balancing at a background I/O priority, but also because you can write faster to the SSD, than you can to the HDDs, which will still limit the balancing. 

 

So, the best option I have to offer is to either use smaller bursts of data, or add a non-SMR drive to the "SSD" list. Or ... add more SSDs. 

 

 

There are some other options you can do to help alleviate how often this will occur, is to change the balancing settings and to bump the priority.

 

In the balancing settings, make sure that the balancing is set to happen "immediately".  If you have the "not more often than every X hours", make sure that the setting isn't too high.  It does default to 12 hours, but that may be too high for your use case.  Setting it to 1 hour or disabling it outright may work better.

 

Also, make sure the balancing ratio slider is set to 100%, as well.

 

 

Additionally, when the software is actively balancing, you can temporarily boost the priority by clicking on the ">>" button at the bottom.  This will boost the I/O Priority during that balancing pass. 

 

If you want to set this permanently, you can do so by editing the config file:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings#DrivePool.Service.config

 

You want the config file, not the JSON file, as that doesn't exist in the release version yet. 

Find the "FileBalance_BackgroundIO" value and set it to "false".  

 

And reboot the system or restart the StableBit DrivePool Service (you can do so by running "services.msc", finding the service and restarting it that way).

i have several Seagate Archive drives and have no slow writes especially when doing sequential write which would be expected with copying large amounts of data to a pool - mine work at between 100MB/s up to 220MB/s - i.e. inside to outside of the disk respectively

 

your ssd should be able to copy to multiple hdd as well as receive data across your lan - mine do and i have duplication on and when copying to the pool across a 10G network

 

maybe you have other issues which is slowing things down?

 

 

I also have SMR drives, and I've DEFINITELY seen the slowdown when copying large files, or a lot of data.  It definitely happens, and it is pretty nasty. No data loss or anything, but it his about 0MB/s for a second or two, and then jumps back up. 

[blondvillain]

The copying/writing to one drive at a time is normal. 

 

Windows' file copy is single threaded, meaning that it only copies one file at a time. And for the most part, this is a good thing. the more you're doing in parallel, the slower it will be overall.  So single threaded is faster, overall. 

 

Ok so that makes sense when it comes to evacuating files into the pool from a drive designated as SSD. But let's take SSD Optimizer out of the equation.

 

If you initiate multiple transfers to DrivePool simultaneously, why wouldn't it route each one to a different drive? This is supposing the SSD Optimizer balancer is NOT in use.

 

Scenario:

 

-SSD Optimizer balancer is NOT enabled.

-You initiate a transfer from F:\, and independent locally installed drive in SERVER1 to a local pool P:\. It begins writing files to the drive in the pool with the most free space available, which as I understand is the default scheme.

-You initiate a second transfer from SERVER2 across the LAN to SERVER1's pool P:\. It begins copying files to the drive with the most free space, which is the same drive the first transfer is currently writing to.

 

In this scenario you now have two independent single-threaded copy operations happening, and contending for IO on the same drive, which in my experience is not ideal. Two simultaneous file transfers will usually take longer than two consecutive file transfers, not to mention the potential for added wear and tear as the write head has to jump around more to accommodate both writes.

 

So why doesn't DrivePool route these two independent transfers to separate drives? This wouldn't be "multi-threaded" as I understand the concept.

 

Consider this scenario where DrivePool isn't involved at all: suppose I have 4 separate hard drives (HDD1, HDD2, HDD3, HDD4). Using Windows File Explorer, if I copy a file from HDD1 to HDD2 and simultaneously copy a file from HDD3 to HDD4, both transfers progress at full speed and Windows tracks them in separate progress windows. In my opinion, this seems like something DrivePool should be able to manage, not to mention seems to be standard operating procedure for Drive Bender.

[Spider99]

"I also have SMR drives, and I've DEFINITELY seen the slowdown when copying large files, or a lot of data.  It definitely happens, and it is pretty nasty. No data loss or anything, but it his about 0MB/s for a second or two, and then jumps back up. "

 

This happens when re writing data not when doing a new copy to free space which is what the OP is doing so things should not slow down

[blondvillain]

"I also have SMR drives, and I've DEFINITELY seen the slowdown when copying large files, or a lot of data.  It definitely happens, and it is pretty nasty. No data loss or anything, but it his about 0MB/s for a second or two, and then jumps back up. "

 

This happens when re writing data not when doing a new copy to free space which is what the OP is doing so things should not slow down

 

Look I'm not here to debate SMR. I'm tyring to get some insight into the workings of DrivePool. Please stop derailing the thread.

[Christopher (Drashna)]

Ok so that makes sense when it comes to evacuating files into the pool from a drive designated as SSD. But let's take SSD Optimizer out of the equation.

 

If you initiate multiple transfers to DrivePool simultaneously, why wouldn't it route each one to a different drive? This is supposing the SSD Optimizer balancer is NOT in use.

 

Scenario:

 

-SSD Optimizer balancer is NOT enabled.

-You initiate a transfer from F:\, and independent locally installed drive in SERVER1 to a local pool P:\. It begins writing files to the drive in the pool with the most free space available, which as I understand is the default scheme.

-You initiate a second transfer from SERVER2 across the LAN to SERVER1's pool P:\. It begins copying files to the drive with the most free space, which is the same drive the first transfer is currently writing to.

 

In this scenario you now have two independent single-threaded copy operations happening, and contending for IO on the same drive, which in my experience is not ideal. Two simultaneous file transfers will usually take longer than two consecutive file transfers, not to mention the potential for added wear and tear as the write head has to jump around more to accommodate both writes.

 

So why doesn't DrivePool route these two independent transfers to separate drives? This wouldn't be "multi-threaded" as I understand the concept.

 

Consider this scenario where DrivePool isn't involved at all: suppose I have 4 separate hard drives (HDD1, HDD2, HDD3, HDD4). Using Windows File Explorer, if I copy a file from HDD1 to HDD2 and simultaneously copy a file from HDD3 to HDD4, both transfers progress at full speed and Windows tracks them in separate progress windows. In my opinion, this seems like something DrivePool should be able to manage, not to mention seems to be standard operating procedure for Drive Bender.

 

 

It can, actually.  But that depends HEAVILY on your pool's configuration. And how the files are copied.

You'd need to either use a multithreaded app (like robocopy) to copy the files, or you'd need to start multiple copy operations. 

 

Specifically, files are placed on the disk with the most available free space. 

So, this would only trigger if all of the drives had roughly the same amount of free space.  And to further complicate this... the files would need to be roughly the same size.  Otherwise, sooner or later, you'd end up copying multiple files to the same disk, since it has the most free space.

 

Additionally, IIRC, the balancer only copies a single file at a time.

 

 

So really, there isn't a good way to facilitate this, without a massive architectural change, and without some sort of performance penalty, too.  Either upfront, or later, due to "excessive balancing" (as a lot of data may end up needing to be rebalanced later). 

 

 

 

 

 

 

"I also have SMR drives, and I've DEFINITELY seen the slowdown when copying large files, or a lot of data.  It definitely happens, and it is pretty nasty. No data loss or anything, but it his about 0MB/s for a second or two, and then jumps back up. "

 

This happens when re writing data not when doing a new copy to free space which is what the OP is doing so things should not slow down

 

I've seen this on a brand new drive, actually.  It happens with the "PMR" buffer gets filled, while waiting to write to the SMR locations, from what I understand.  For smaller transfers, you shouldn't see this, but when transferring 100+GB of data in one go, then yes, you can and likely will run into this. 

[AlexL]

Hi,

I also had issues with SSD not being emptied in a timely fashion (as are many users it would seem, judging by the amount of posts in this forum alone) so my only solution was to put a few SSD's into a Storage Spaces volume thus having a large enough capacity to begin with. Still takes time for the SSD optimizer to move files but having a larger faster blurb did it for me.

 

BR  

[Christopher (Drashna)]

Hi,

I also had issues with SSD not being emptied in a timely fashion (as are many users it would seem, judging by the amount of posts in this forum alone) so my only solution was to put a few SSD's into a Storage Spaces volume thus having a large enough capacity to begin with. Still takes time for the SSD optimizer to move files but having a larger faster blurb did it for me.

 

BR  

 

 

The emptying is controlled entirely by the balancing settings.  Chances are that your balancing settings are configured in such a way that hamper the effectiveness of the balancer.  

 

Namely, setting it to "balance Immediately", unchecking the "not more often" box, and setting the ratio to 100% should make it VERY aggressive about emptying the drive.