Readstriping x2 vs x3 duplication?

[thepregnantgod]

I'm curious if setting my duplication levels to higher than x2 would increase the read striping.  I have 30 drives attached to my file server but most of the files are very large .iso rips (sometimes 30gb or higher).

 

So I'm betting that it can only read that one file at a time?  Or does it magically somehow pull the data from 2 different copies at the same time?  And...if so, would providing a third by increasing my duplication ot 3 help?

[Christopher (Drashna)]

It most certainly wouldn't hurt.  

 

Depending on the situation, it will either read from multiple copies of the file at the same time and boost the read speed (especially for large, sequential reads).  Otherwise, it can/will read from the least active disk.  This depends heavily on what the system is doing at the time. 

 

You can read more about the feature here:

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options

 

 

 

But adding another copy will increase the odds of a less active disk, or increase the number of disks to read from simultaneously. 

[thepregnantgod]

So, Drashna, to confirm, Drivepool can access two (perhaps 3 or more) copies of a particular file and then push them to whatever PC is requesting them and recombine them on the other end?

 

I RAID stripe my OS so I understand that a file larger than the block size is actually written in parts across all the drives in the striped RAID.  But with Drivepool, the file is not broken into parts.  These files are full individual copies.

 

Can you explain a bit more?

[Christopher (Drashna)]

Sort of. 

 

Specifically, when accessing a file, if there is no disk that is on a faster bus (eg USB vs SATA vs etc) and no disk is working, it will grab the contents from all the disks, and read different chunks of them into memory and piece them together.  

 

It does this in 1MB chunks by default (IIRC).  So for a 6MB file, it would essentially break it up into 6 sections to read from, and for a x3 duplicated file, it would read the first three chunks from the different disks, and repeat, as needed.  Spreading the IO load out between the disks, and potentially, getting a higher read speed. 

(specifically, we can grab a section of a file, at a time, rather than having to read the entire file into memory... and this is what Windows does normally anyways). 

 

This is essentially how a RAID array works, but rather than letting the "controller" (in our case, the driver) determine how to handle this, the data is spread between the disks to start with. 

 

 

 

However, unlike RAID, if one of the disks becomes busy from other sources while the read is occuring, we can switch to the other disks, or to a single disk, on the fly.  

 

 

Actually, Alex has a nice, long post that goes over this too:

http://blog.covecube.com/2013/05/stablebit-drivepool-2-0-0-256-beta-performance-ui/

 

 

 

 

The caveat here is t hat even with all the optimization, you are probably not going to get as good performance as RAID. There is some additional performance overhead for how we operate, and because of the block based layout.  The tradeoff here is flexibility.