DrivePool + Primocache

[Jaga]

I recently found out these two products were compatible, so I wanted to check performance characteristics of a pool with a cache assigned to it's underlying drives.  Pleasantly, I found there was a huge increase in pool drive throughput using Primocache and a good sized Level-1 RAM cache.

This pool uses a simple configuration: 3 WD 4TB Reds with 64KB block size (both volume and DrivePool).  Here are the raw tests on the Drivepool volume, without any caching going on yet:

 

After configuring and enabling a sizable Level-1 read/write cache in Primocache on the actual drives (Z: Y: and X:), I re-ran the test on the DrivePool volume and got these results:

 

As you can see, not only do both pieces of software work well with each other, the speed increase on all DrivePool operations (the D: in the benchmarks was my DrivePool letter) was vastly greater.  For anyone looking to speed up their pool, Primocache is a viable and effective means of doing so.  It would even work well with the SSD Cache feature in DrivePool - simply cache the SSD with Primocache, and boost read (and write if you use a UPS) speeds.  Network speeds are of course, still limited by bandwidth, but any local pool operations will run much, much faster.

I can also verify this setup works well with SnapRAID, especially if you also cache the Parity drive(s).

I honestly wasn't certain if this was going to work when I started thinking about it, but I'm very pleased with the results.  If anyone else would like to give it a spin, Primocache has a 60-day trial on their software.

[Christopher (Drashna)]

Very nice!

[B00ze]

Good day.

I think CrystalDiskMark bypasses the Windows cache when it runs its tests, but cannot bypass PrimoCache. So while PrimoCache looks impressive, you should really compare with something that doesn't bypass the windows cache, if you want to know how much faster PrimoCache is compared to native caching...

Regards,

[Jaga]

I didn't really do it to compare speeds, but rather to show that a popular, robust, and established caching program works with DrivePool.  And it's one I highly recommend (I run it on all my workstations, laptops, and server).  The only caveat in the case of using it with a Drivepool, is that usually pool drives are very large, so you need a lot of RAM to get even a small hitrate on the cache.  But for writes, most recently accessed files and frequent files, it's quite effective.

I ran a similar test with Anvil Pro, despite it being geared more against SSD testing (which really is how you should benchmark Primocache).  Here's the test with Primocache paused:

 

And the test with Primocache re-enabled:

 

It shows a ridiculous increase in speed, as you'd expect.  The read/write speeds for both tests are exactly where I'd expect to see them.  And since Windows' cache isn't all that aggressive (nor does it auto-scale well), I would not expect to see any difference using a benchmark that was intended to use it.  Anvil Pro may - I don't know.  Certainly the Windows cache wouldn't have much of an impact once you start scaling the test over 1GB.

Feel free to run the test yourself, with whatever benchmark software you want.  Primocache has a free 60-day trial license with full features so people can test it themselves.

[Jaga]

As a side note:  I've been using Primocache for around 6-7 years, since it was called Fancycache.  Windows' cache can't even begin to compare.  How do I know this for certain?  The server that Drivepool is being benchmarked on in the above examples has 32gb of RAM, running Windows 7, with a 5400 RPM system drive.  I've explicitly set Windows' system cache to allow it to expand well past normal limits.  Booting the system and running things like Plex take absolutely forever, but doing the same thing with Primocache installed takes *far* less time.  It almost runs at the same speed my 4.7ghz workstation with a 512GB Samsung Pro SSD runs.  That's how good Primocache is.

But hey - I'm not here to pimp Primocache, just to let Drivepool users know they have an option if they want to speed up their pools.  If you don't believe my numbers and opinions, do as I suggested and test it out yourself.

[B00ze]

Hi Jaga.

Comparing speeds with PrimoCache and then without ANY cache is not very useful, especially if you tell CrystalDisk to use a tiny 500MB file. I replied because I wish you would post numbers that compare the Windows cache VS PrimoCache, as I have been interested in PrimoCache. I used to run PowerCache (I think it was called) on Amiga, and could set the number of sets and the number of lines for the cache for each disk, so I was able to optimize depending on what kind of data was on the disks, and it was quite efficient. It was, like FancyCache, a sector-based cache, and I think it is a bit better than what Microsoft does, which is a FILE cache. BUT, since absolutely everything in NTFS is a file, using a file cache should not be all that bad; plus, if Windows does like BeOS, it can run programs directly from cache without first having to copy them somewhere else in RAM. Also, as far as I know, you can't "set Windows' system cache to allow it to expand well past normal limits" - In Win7 at least, it will used-up ALL available RAM by default (at least it does on my 32GB system, and I haven't touched any cache parameters.)

I'd use PrimoCache almost immediately if I could tell it to do like Windows does - Grow to use all available RAM, then shrink when something needs the RAM. I'm not too keen having to specify in advance how big the cache will be. What happens if we use BOTH PrimoCache and Windows Cache? I might be interested in a small PrimoCache, below the native cache (but really, what I want is a cache that grows and shrinks, and that does write-back...)

Regards,

[Jaga]

 

Also, as far as I know, you can't "set Windows' system cache to allow it to expand well past normal limits" - In Win7 at least, it will used-up ALL available RAM by default (at least it does on my 32GB system, and I haven't touched any cache parameters.)

Sure you can.  Windows 32-bit program here.  64-bit here.  Just don't change anything using the 32-bit interface after setting limits using the 64-bit program, or it might fubar the settings.

 

 

I'd use PrimoCache almost immediately if I could tell it to do like Windows does - Grow to use all available RAM, then shrink when something needs the RAM. I'm not too keen having to specify in advance how big the cache will be. What happens if we use BOTH PrimoCache and Windows Cache? I might be interested in a small PrimoCache, below the native cache (but really, what I want is a cache that grows and shrinks, and that does write-back...)

Primocache can't do that, but their other product Primo RAMdisk can.  You set a maximum limit for it, and it only takes as much RAM as the RAMdisk is currently using (I have it installed and working that way myself).

It's a good suggestion for Primocache however - I may toss it at the main developer over there and see what they think of a min/max limit on the cache so it can dynamically adjust.

Primocache exists just fine right alongside the Windows cache.  Since Windows isn't aggressive in file caching (at all), it typically uses very little RAM for files it reads.  And it doesn't have a write cache (I've never heard of anyone seriously using the write-back feature and getting good results).  That's where Primocache picks up - it has a much more robust kernel-level cache that will pick up where Windows leaves off, they exist in perfect harmony.  If a file is in Windows' cache, Primocache doesn't take any action, it just lets Windows deliver the content.  If it's not in Windows' cache, Primocache caches the blocks being requested (it's block level, not file) and delivers the content.  Since it is kernel level, it can also be told to pre-fetch the contents of the cache at last shutdown during boot time, to speed up the system before Windows has even loaded.  My boot times (with the RAMdisk turned off) are around 6 seconds, and I'm not using a NVMe.

Primocache's write cache is quite robust as well, and highly recommended with a UPS on the machine.  Both of my benchmarks above are with the write cache on, and when using a large L1 (RAM) cache for both read/write with large block size.

Even if you only have 1GB to throw at a L1 cache (or a spare SSD to use as a L2 cache against spinner drives), it can dramatically speed up almost any system.

I rebooted my server just this morning, so the cache hit rate isn't as high as it would normally be (around 93%), but this gives you a good look at how I have my boot volume cache configured, and at the Primocache interface on a live machine:

 

 

If you know of a benchmark program that works at the file level (and not sector or drive level), let me know what it is and I'll re-run the benchmarks at whatever file size you think is good.  It'll be interesting to see how Drivepool responds in both cases.

 

p.s.  Thanks @Christopher (Drashna) & @Alex for allowing us to discuss non-Covecube products on the forums here!   

[B00ze]

Hey!

Thanks for discussing this. You have me interested. When I have time I will go to their forums/feedback and ask about a dynamic-size cache. They will probably tell me it cannot be done, lol, but if they did manage it for their RAMDisk then why not with the cache? Even with a fixed-size, it /could/ be good to have a smallish PrimoCache sitting underneath the Windows cache, especially with write-back (you are correct, Windows' write-cache sucks.) I see you can group disks and create separate caches, that's good - I probably would not want to cache my fast-enough SSD, but the DrivePool is something else! It took an HOUR to 7-zip unpack 1.2 million small files the other day, Gods it was slow!

About CacheSet: Oh I have this already, but here are the Win7 defaults: Current minimum working set size: 1048576 Current maximum working set size: 1099511627776. That is, 1TB maximum size by default, so no need to touch it ;-)

For disk benchmarks, we need something like AnandTech's Storage Bench. I checked AIDA but I don't see what I want in there. Maybe you can try IOZone or Sandra ?

Best Regards,

 

[Jaga]

I've used SiSoft Sandra many years ago, so that's one option.  Don't know if it's sector/drive or file cached, but a little investigation and testing might tell.  Never used IOzone, but it might be interesting to run too.  And I'm fairly certain Christopher and Alex would like to see the results, if for no other reason than morbid curiosity.

My Windows 7 Ultimate 64-bit server (the one in the tests) has 32GB of RAM, and with just the Windows cache it ran abysmally slow.  It almost got to the point of irritation, so I broke down and bought a copy of Primocache for it and it was night-and-day.  I don't know how Windows 7 32-bit could address a 1TB cache, perhaps that's the x64 window you're seeing.  Mine with only the Windows cache never used up more than ~2GB total RAM, even after leaving it on for weeks serving up Plex music/movies.  So it's not nearly as aggressive as it should be.

If you get around to installing Primocache, let us know your results.  And/or ask for recommended settings over there - if you use the same username I'll see and reply.

[fattipants2016]

Will primocache store file/folder directory tree information so that every darn disk in my computer won't spin up when I open Windows Explorer?

I'd say DrivePool's own SSD optimiser is an effective alternative for this software already, but I would kill for ^^ this functionality.

[Christopher (Drashna)]

 

p.s.  Thanks @Christopher (Drashna) & @Alex for allowing us to discuss non-Covecube products on the forums here!   

Well, it's storage related, and may be something that helps out others.  
Much like how we have a number of threads discussing SnapRAID and the like. 

 

Will primocache store file/folder directory tree information so that every darn disk in my computer won't spin up when I open Windows Explorer?

That would be a real kicker, and seriously help with DrivePool.

 

 I'd say DrivePool's own SSD optimiser is an effective alternative for this software already, but I would kill for ^^ this functionality.

Yes and no.  For modifying existing data or reading data, PrimoCache and the like may be very useful. Much more so than the SSD Optimizer.

[Jaga]

 

Will primocache store file/folder directory tree information so that every darn disk in my computer won't spin up when I open Windows Explorer?

Will it cache that information?  Yep.  Anything that exists on the drive can be cached.  Will it leave the drive asleep and still deliver the information?  Probably not, since it allows the Windows cache to operate alongside it, and I'd suspect it would wake the drive up on any kind of ping for information.  Plus - it's impossible currently to force Primo to cache what you want.  It caches what it sees as the most accessed information.  IF that information happens to be your file/folder information for the drives and Drivepool, then some or even many of them might not wake up.

However - from what I understand, the single greatest wear-and-tear operation on a spindle drive, is spinning up from sleep repeatedly.  A long time ago I used to allow mine to sleep, but having learned this since.. I stopped allowing them to spin down.  The savings on power between a spinning idle drive and a sleeping idle drive aren't large, and I value lifespan over minimal cost savings.  You might want to review all of your drives and see if the extra energy use is worth the additional wear on them.

 

 

I'd say DrivePool's own SSD optimiser is an effective alternative for this software already, but I would kill for ^^ this functionality.

Drivepool's SSD optimizer is definitely not a replacement for Primocache's L2 caching.  The closest it would come is acting as a pseudo-write-buffer for the DrivePool, which Primocache's L2 SSD caching could do just as well, since it works for both read and write caching at the block level.  Drivepool's SSD feature is a temporary and fast adding-files-to-the-pool-only area, whereas Primocache's L2 SSD cache is a true read/write buffer for any kind of drive installed on a computer.  And, it can persist in-between reboots of the machine.

They are very different animals, both trying to increase speed, and both having their uses in different ways to different effect.  DP's SSD feature targets the pool for writes only, Primocache's targets any drive(s) in the system for reads and writes.  I wouldn't recommend using both at the same time on the Pool.  While Drivepool's SSD feature is definitely a nice addition to the suite, Primocache's L2 effect on the system/pool would be much more effective, but also cost more money as you need to buy a license for it.  Separately, Primocache also offers L1 RAM caching.

As an example:  when I rebuild my storage pool this summer, I'll be dropping in a rather large SSD to use as a read/write buffer with Primocache on that server, since the L1 RAM requirements to cache against a large storage Pool would just be way too big - 2TB of RAM just isn't going to happen.  But a fast 2TB SSD as a L2 cache in Primo could easily handle caching on a 50TB Drivepool (4% of total data covered for reads, all writes covered).  I *could* simply use DrivePool's SSD optimizer, but that wouldn't help with read caching, and I've already bought the license for Primocache on that machine.

Definitely room for both products, depending on what you want/need and how much money you're willing to put towards the solution.  Ultimately as far as caching and pool speed goes without considering cost, Primocache (using the same SSD) would be a more encompassing solution than DP's SSD optimizer.

[fattipants2016]

 

Drivepool's SSD optimizer is definitely not a replacement for Primocache's L2 caching. 

I don't know if it will be worth $30 in my use-case (archival data) but I'll definitely give it a whirl and see if it knocks my socks off.

Do you recommend caching the physical disks, or the pool?

[Jaga]

 

Do you recommend caching the physical disks, or the pool?

The physical disks.  Drivepool's pool doesn't have any real sectors to cache, which is what Primo works with.  The three drives in my screenshot above (Z, Y, and X) are my pool drives, so I told Primo to cache them physically.  If you're just going to setup read-and-write caching on a L2 SSD, then you don't need to have the server on a UPS for power protection. 

[Jaga]

 

For disk benchmarks, we need something like AnandTech's Storage Bench. I checked AIDA but I don't see what I want in there. Maybe you can try IOZone or Sandra ?

I briefly tried IOZone, but it's command line only, doesn't create graphs in the XLS file, and the data is more than I'd care to use in a casual comparison.  Trying Sandra and unchecking bypass windows cache showed throughput around 4x that of normal Windows caching.  I'm not sure I trust that result however, since it does things like testing disk position/etc.  It doesn't just have a raw throughput (read/write) for different queue depths and test file sizes.  

[B00ze]

Good day Jaga.

Arg, so IOZone produces complicated CSV files? Darn, it looked perfect, all the website said it does is call standard disk stuff, not trying to measure position on disk or bypass cache. Sandra also looked promising. It's so difficult to find a benchmark for disks that doesn't try to get the disk's raw performance. Well who knows, maybe it is 4 times faster with PrimoCache enabled! When I go to their forums (it's on the todo list) to ask some questions, I'll ask if they know of a benchmark we can use.

Thanks!

[fattipants2016]

 

The physical disks.

I see you can configure the delayed write option to store the data 'indefinitely' in level 2 cache, but I don't see anywhere that you can schedule how often it flushes to disk.

If I could store all of my file changes in level 2 cache, and flush them to disk on a schedule once a night, it would be a perfect replacement for the way I'm using the SSD optimizer.

I'm trying to mitigate changes to my archival drives between updates of my SnapRAID parity data, but if primocache had this scheduling feature it'd be even better.

[Jaga]

 

I see you can configure the delayed write option to store the data 'indefinitely' in level 2 cache, but I don't see anywhere that you can schedule how often it flushes to disk.

If I could store all of my file changes in level 2 cache, and flush them to disk on a schedule once a night, it would be a perfect replacement for the way I'm using the SSD optimizer.

I'm trying to mitigate changes to my archival drives between updates of my SnapRAID parity data, but if primocache had this scheduling feature it'd be even better.

Hey @fattipants2016

It's actually rather easy to do, I have my L2 SSD cache (in Primocache) set to flush writes every 30 seconds, but you can configure that interval to whatever you want.  24 hours would be 86400 seconds, you'd put it into the Latency box in the following interface for Primo:

The L2 cache is persistent, even across reboots (you can enable/disable that at will).  It will hold writes as long as you tell it to, or not at all.  So from 0 (disabled write caching) up to whatever interval you want (I haven't tried super-long delays yet).  It'll flush those writes to the disk it's caching automatically.  I just tested it with the value of 86400 (1 day) and it took the setting without complaint.  I suspect the timer 'resets' and begins counting down after a manual flush (you can do it with a button press in the UI), so you can determine what time of day it flushes easily.

 

[Julius]

I've been an avid (paying) user of SoftPerfect RAM Disk. I wonder if I could set aside some RAM for my DrivePool using that. Any ideas?

Generally I have all my user profiles and portable apps on a RAM disk. That gets stored away at shutdown (and at a set interval, like with primo) and re-opens at startup.

[Jaga]

There should be no reason you couldn't use a RAM disk as a pool member, provided it was basic drive (emulation) with a NTFS format.  You might want to force a manual save-to-disk-image before every shutdown, just for diligence.  However due to the limited space on them, you'd also want to limit what files the pool was able to balance onto the RAM disk.  OR, simply use the RAMdisk as a SSD Cache plugin drive, to speed up delivery of files to the pool.

I stopped using Softperfect years ago, switched over to Primo RAMdisk instead.  I find it to be a more robust solution that's also more flexible.

[moskeale]

Hey @Jaga - I know it's been a couple years since you checked in on these forums but in case you get this message ....

I used Primocache a couple years ago with Drivepool but at the time was suffering a lot of BSODs and suspected it could be a cause of conflict, so I stopped using it after my system M.2 drive failed and I had to redo everything.  In hindsight now I believe my conflict was actually "HFS+ for Windows", (besides the point).... so trying to reimplement my Primocache usage now.

Just curious if you're still using Primocache and Drivepool together, read & write cache, and if you've had any issues in the past 3 years?

My use case:
Drivepool: 4 x 5TB drives, 2 x 6TB drives, (with full file duplication) ..... adding 2 x 1TB SSDs for SSD Optimizer (write caching)
Primocache: read cache only, L1: 4GB RAM, L2: 250GB on M.2 partition

(I'm a video editor and animator and I handle a lot of big projects).

[Christopher (Drashna)]

On 11/30/2021 at 9:29 AM, moskeale said:

In hindsight now I believe my conflict was actually "HFS+ for Windows", (besides the point).... so trying to reimplement my Primocache usage now.

we've run into a few people that have had issues with the HFS driver for windows causing BSODs, and a good number of people using primocache with StableBit DrivePool without issues. 

[gtaus]

FWIW, if anybody is still interested in this topic, I tried PrimoCache with DrivePool on my home media server, and the transfer speeds did indeed show significant improvement within that computer. However, most of my workflow is from transferring files from client computers into my host server. In that case, PrimoCache has NO effect at all. PrimoCache does not work with network drives, which is how my client computers see the DrivePool volume on my server. So, transferring files from my client computers to DrivePool on my server has absolutely no improvement in transfer rate with PrimoCache. 

I have found my better option is to use the SSD Optimizer as a frontend cache for DrivePool, which allows me to transfer my files from a client computer into the SSD frontend of DrivePool on the server. Then the SSD will rebalance the files as needed on to the archive HDDs in the background. Of course, file transfers are still limited by your network speed, but having the SSD frontend on DrivePool pick up the files is usually much faster than writing to the archive HDDs directly over the network. 

I contacted PrimoCache support about this issue, and they verified that PrimoCache does not work with network drives. Since network transfers is the bottleneck in my system, I ended up not purchasing PrimoCache although I do think it's a very good solution for other setups.