Not using all drives

[LindsayCole]

Hello,

 

I currently have a massive transfer underway, 12TB via three write threads at a time (using RichCopy).

 

I am seeing only 4 drives (out of 16) getting used, read / write speeds are average at best (50-125 MB/s).

 

Is this the expected performance on this? I expected faster being that source is all 15k RPM drives, and that destination is multiple drives.

 

Is it normal for it to only use a few of the drives at a time?

 

Is it normal to only be getting a single drives throughput at a time, even though I see it putting files on two different drives at once?

 

Thanks!

[Christopher (Drashna)]

This is absolutely normal.

To be clear here, you are not going to see RAID like speeds here. StableBit DrivePool is a file based solution, and because of that, we don't have the optimization that a block based solution (RAID) will see.  We do have some optimizations for read speeds that may get closer to those speeds, but....

 

 

Also, most file copy utilities only copy one file at a time. That means that you're only writing to one disk at a time. 

To get faster speed, you'd need to write in parallel. But that will also be bottlenecked by your network connection. 

 

 

 

Specifically, your speeds look about right for "normal" drives (aka, 7200RPM or lower HDDs). 125MB/s is the max you're going to get from gigabit networking without using teaming (on both systems, and with a switch that supports it), or using 10gig networking.

However, the 50MB/s looks a bit on the slow side, but that depends on the disks in question (and writes will always be slower than reads).

 

 

StableBit DrivePool's default file placement is to put files on the disk with the most free space (absolute, not percent).  This means that large drives will be used first, and then it will later start using other drives as well.

 

There is an option balancer that changes that strategy and uses all disks equally. If you want to use that, grab the "Disk Space Equalizer" balancer.

If you want to maximize that speed, you may want to look into the "SSD Optimizer" optional balancer (at least temporarily). It lets specify one or more drives (usually SSDs) to be the "landing zone". It writes to those drives, and then balances the data off of the drives. 

 

Here is a link to both balancers:

https://stablebit.com/DrivePool/Plugins

[LindsayCole]

Hey Drash, thanks for the response.

 

Let me give you a clear view of my topology.

 

I have a server, with 20TB of internal storage. I have a SAS card, flashed to IT mode, connected to a SAS enclosure with another 20TB of storage. So networking is out of the picture.

I am using RichCopy (which is doing three write threads in parallel). My hopes were, that because DP would be writing to multiple disks (from a faster source) that I would get increased speeds since each individual drive is able to be written to on its own.

 

I was most definitely making assumptions when I was doing this setup, but thats how I learn best

 

If I used a balancing plug in, say Disk Space Equalizer, do you expect that speeds would increase? I just want to set my expectations properly

With the SSD Optimizer, does it only work with SSD's? Or could I configure it to work with some 15K RPM drives as well?

 

Also, its hard to say for sure, but when I see the drops at 50MB/s, it could also be smaller files, which could easily bottleneck the transfer.

[Christopher (Drashna)]

Ah, yeah, that's a lot of storage, and I can sympathize (with my ~42TBs of pooled storage) (and those IBM ServeRAID m1015 cards are nice, aren't they ).

 

If RichCopy is doing the writes in parallel, it really depends on the disks in the pool, and how much free space they have (again, measured in absolute size, not percent).

If they're all about equal, then they should, in theory write to each disk separately and speed the process. But still, that depends.

 

 

As for the SSD Optimizer balancer, nope, doesn't matter. You could use a slow 5400RPM laptop drive, if you really wanted to... But most people use SSDs for speed and not 15k RPM drives. So that's why it has the name. So you absolutely could use the 15k RPM drives as the "SSD" drives here. In fact, I was going to recommend that.

 

 

Additionally, in case it helps (sorry for not mentioning this sooner, as well), but if these drives are all on the same system, you could just "seed" the pool. 

http://wiki.covecube.com/StableBit_DrivePool_Q4142489

StableBit DrivePool can absolutely use RAID drives, as long as you're not using Dynamic disks for the RAID (but I suspect you're using hardware RAID here). Just add the existing RAIDed drives to the pool, "seed" the pool, and then you're set. 

And if you want to migrate the data off of the RAID drives, you could use the "Disk Usage Limiter" to do this. However, this may be slower than a straight file copy, as balancing is run in a background IO priority. Which can be changed: http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings (set FileBalance_BackgroundIO to "False")

 

 

And if you're moving a lot of small files, then yes, that can definitely happen. That's the thing that sucks about small files.... they really kill the transfer speed. But if that's 50MB/s for small files, that's not bad.

[LindsayCole]

Ah, yeah, that's a lot of storage, and I can sympathize (with my ~42TBs of pooled storage) (and those IBM ServeRAID m1015 cards are nice, aren't they ).

 

I love them!

 

If RichCopy is doing the writes in parallel, it really depends on the disks in the pool, and how much free space they have (again, measured in absolute size, not percent).

If they're all about equal, then they should, in theory write to each disk separately and speed the process. But still, that depends.

 

I did some testing, with only two threads, I saw a speed increase to about 175-325MB/s. Seems like it may have been an issue with RichCopy.

 

As for the SSD Optimizer balancer, nope, doesn't matter. You could use a slow 5400RPM laptop drive, if you really wanted to... But most people use SSDs for speed and not 15k RPM drives. So that's why it has the name. So you absolutely could use the 15k RPM drives as the "SSD" drives here. In fact, I was going to recommend that.

 

Perfect, I'm leaning towards that.

 

Additionally, in case it helps (sorry for not mentioning this sooner, as well), but if these drives are all on the same system, you could just "seed" the pool. 

http://wiki.covecube.com/StableBit_DrivePool_Q4142489

 

StableBit DrivePool can absolutely use RAID drives, as long as you're not using Dynamic disks for the RAID (but I suspect you're using hardware RAID here). Just add the existing RAIDed drives to the pool, "seed" the pool, and then you're set. 

And if you want to migrate the data off of the RAID drives, you could use the "Disk Usage Limiter" to do this. However, this may be slower than a straight file copy, as balancing is run in a background IO priority. Which can be changed: http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings (set FileBalance_BackgroundIO to "False")

 

I saw that, I needed to move the data out of a .vhdx mounted storage space- so the seeding wouldn't have worked well for me.

 

And if you're moving a lot of small files, then yes, that can definitely happen. That's the thing that sucks about small files.... they really kill the transfer speed. But if that's 50MB/s for small files, that's not bad.

 

That definetly seems to have been the issue.

 

Have you had any experience yourself, using SnapRaid with DrivePool to acheive parity?

 

I'm at a critical spot with the build, where I need to make the decision- and I don't want a half assed solution

 

It seems like FlexRaid will do everything I want, but I really like DrivePool - but FlexRaid seems to really complicate things. I want to set and forget ;-)

[Christopher (Drashna)]

That's definitely a significant performance increase! And if is the case, then yeah, it sounds like there may be an issue with RichCopy then. Not sure if there is anything we can do in that case.  Though, if you want to enable file system logging and reproduce, we could look into it. 

 

And yeah, recovery from the VHDX files would make that much more complicated, so seeding probably wouldn't have worked.

 

 

 

As for parity, we have a number of users that have set up FlexRAID and SnapRAID to achieve parity with StableBit DrivePool. Basically cheating, by using the simplicity of StableBit DrivePool for the pooling and using Flex/Snap-RAID just for parity.

 

The thing is that you will want to enable the "Ordered File Placement" balancer. It's a stand alone balancer, or it is integrated into the SSD Optimizer as an option (due to a weird interaction between the two).

 

We have a few threads about these, which I've gathered and linked together:

http://community.covecube.com/index.php?/topic/52-faq-parity-and-duplication-and-drivepool/

The post with the links is at the bottom.