Jump to content
  • 0

Automatic duplication to improve read speeds?


Jason Simone

Question

Hi, I am currently considering my options for a new storage volume and I have just run across this product. I am trying to decide between implementing DrivePool or going with a more traditional RAID solution. Something that attracts me to software or hardware mirroring is the fact that I expect to have many drives and the possibility that I can get better I/O performance through mirroring and striping over many columns.

Regarding DrivePool performance, I do see claims on the website that data duplication will improve performance by reading from more than one source at a time. However, it seems to me like the primary purpose of duplication is for data protection / drive failure. What I was wondering is, whether DrivePool can also use additional space on parallel drives to store additional mirrors and improve performance even more? For example, I might have a folder that I don't need a duplicate of, or I don't need more than a single redundancy (for failure protection), but I would like it's read speeds to be optimized as much as possible. So if I have free space on more than two drives, will DrivePool be able to automatically stripe/cache additional copies of certain data and then read that data from several drives when it is requested? Or if this is not currently an option, has this been a requested feature / on the roadmap?

Thanks for any help!

Link to comment
Share on other sites

4 answers to this question

Recommended Posts

  • 0

DrivePool doesn't use write striping, though it will take advantage of duplication (if enabled) to perform parallel writes and to perform read striping.

It will not dynamically enable additional duplication; e.g. if you've set a particular folder to x3 duplication then it will write instances of each file in that folder to three drives, no more, no less. I don't know if dynamic duplication is a requested feature (or how feasible it would be); you're welcome to use the contact form to send such a request to the developer.

Link to comment
Share on other sites

  • 0
On 12/30/2020 at 1:55 PM, Jason Simone said:

I am currently considering my options for a new storage volume and I have just run across this product. I am trying to decide between implementing DrivePool or going with a more traditional RAID solution. Something that attracts me to software or hardware mirroring is the fact that I expect to have many drives and the possibility that I can get better I/O performance through mirroring and striping over many columns.

In the past, I have used hardware RAID arrays, then I moved to Windows Storage Spaces (software ~RAID), but now I use DrivePool for my home media storage. Hardware arrays can be very fast and you can get great I/O performance. Windows Storage Spaces uses striping and, depending on the pool type you setup, you can get fast I/O performance. However, if you have drive failures in either hardware RAID, or Windows Storage Spaces, data recovery can be next to impossible because they strip their data over many drives in the pool. Despite having 2 drive failure set on my Storage Spaces, I lost 1 HDD and it took down my entire Storage Spaces pool of 26 drives - lost everything.

DrivePool is limited to the read/write speed of the HDDs you use. If you don't have duplication on your pool/folder, then you cannot take advantage of read striping. However, I recently added a SSD to the front end of my DrivePool and set it to cache 100GB of data. So now, I can use my fast SSD cache for DrivePool writes, and as a temp directory for read/writes for certain programs I use. I am now getting faster I/O performance than I ever got on my older RAID and Storage Spaces systems.

If you really want to step up to higher performance, then I would suggest a product called Primo Cache which can use both system RAM and SSD as cache for your system. I think they still offer a free trial period. If I was running a server with lots of I/O requests, I would certainly consider purchasing Primo Cache. However, just adding a SSD to the front end of my DrivePool more than meets my needs for my media home storage.

On 12/30/2020 at 1:55 PM, Jason Simone said:

What I was wondering is, whether DrivePool can also use additional space on parallel drives to store additional mirrors and improve performance even more? For example, I might have a folder that I don't need a duplicate of, or I don't need more than a single redundancy (for failure protection), but I would like it's read speeds to be optimized as much as possible. So if I have free space on more than two drives, will DrivePool be able to automatically stripe/cache additional copies of certain data and then read that data from several drives when it is requested?

No, DrivePool does not create more than 1 copy of any file unless you tell it to duplicate the pool or that folder. However, the power of DrivePool is that you can designate which folder(s) you want duplicated and you don't have to duplicate the entire pool. So if you have certain folders that you will need higher I/O performance, you can just set them to 2X or higher duplication and take advantage of Read Striping. (I don't know if DrivePool would take advantage of 3X or 4X duplication for read striping from 3 or 4 drives.)

In my case, if I need to use a higher performance I/O cache for working programs, I will flush my 100GB SSD cache in DrivePool and then direct my temp directory back to DrivePool. 100GB of SSD cache is more than enough for me, however, Primo Cache would be even faster as it can use system RAM as well as your SSD.

I find DrivePool more reliable than my older hardware RAID or Windows Storage Spaces solutions. Data recovery from a drive failure is better with DrivePool. Adding a SSD to the front end of my DrivePool makes it faster than I ever got with my other systems. I only add 2X duplication on certain folders I don't want to have to rebuild from backups, but most of my media storage DrivePool is only at 1X and that saves me lots of money on drives. DrivePool just works better for me.

 

 

Link to comment
Share on other sites

  • 0

@Waltski, the DrivePool SSD optimizer plugin is only a write cache, so it would have no effect at all on your streaming (reads) movies to your TV. I have not yet moved to 4K files, but my 1080p files stream at about 1 MB/s or less, so Read Stripping at 2X would not make any difference for me. I have all my media folders set to 1X.

On my home media system, my bottleneck is first in the Amazon Fire TV stick with limited RAM cache and low powered CPU, then second is my home wifi which has its limits. DrivePool is not on the list of things I consider to be a bottleneck on my media system because DrivePool can transfer files over my wired network at 30 MB/s or higher. But wifi is much slower and if many devices are accessing the wifi at the same time, things just get less reliable for streaming. I start to see buffing problems on my Amazon Fire TV stick with 1080p files larger than 16 GB, but that is due to my Fire TV stick and wifi limits, not DrivePool.

I have a media center computer hooked up in the far room, and it can stream much larger movie files via wired ethernet. But that computer also has much more power than my Amazon Fire TV stick.

I am not familiar with AppleTV 4K. Is that a wired device or wifi device? When you stream movies to your AppleTV 4K, you can use the DrivePool GUI to monitor the speed of your outgoing files. In my case, with the 1080p files, after the initial burst to fill the cache on my Fire TV Stick, the transfer rate drops to less than 1 MB/s. Since I know my DrivePool can transfer files at 30 MB/s from computer to computer over the network, I am assuming my Fire TV Stick can only accept that 1 MB/s speed for incoming data. 

I have a friend that was big time into Plex, and he told me that some game machines allowed a person to download/pre buffer a movie on the game machine's HDD and then you just play the move back locally from that HDD. That was a great option for sharing files over a slow internet connection. They would queue up the transfers the night before and then watch all the movies locally the next day. Is something like that possible with the AppleTV 4K?

At any rate, use the DrivePool GUI to see what the transfer rate to your AppleTV 4K device is pulling and then you will probably see that a 1X DrivePool is more than enough to keep up with it. If you can, please post what DrivePool GUI transfer speed you see on your streaming 4K movies. I would like to know as a comparison to the 1080p files I am currently streaming. Good luck.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...