Drivepool SSD + Archive and SnapRAID?

[fly]

So I'm currently using Drivepool with SnapRAID for backup.  For SnapRAID to run quickly, I've turned off the regular balancers and instead just rely on file placement rules to keep everything in balance (I'm mostly just adding and rarely deleting).  I've started adding a couple of SSD drives that I'd like for everything to land on, then using a FIFO type policy move things to magnetic drives as the SSDs fill up.  What is the best way to accomplish this without the balancers shuffling around data on the Archive drives?

[Christopher (Drashna)]

The SSD Optimizer balancer should do this, actually. Just disable all of the other balancers if you want, which is what we recommend with that balancer, anyhow. 

Just read the notes for it, if you want to use file placement rules too.

 

[fly]

Excellent.  Thanks!

[Christopher (Drashna)]

Welcome!

[fattipants2016]

I do exactly this. Be sure to check the 'leave the files where they are' option in the SSD optimiser and it will leave the files on your achive disks alone.

There's no FIFO, though, you'll need to schedule or define triggers for balancing, at which point Drivepool will empty your SSD's onto your archive disks.

[fly]

So without FIFO, when the drive(s) get full, it dumps all of the contents to the archive drives?  Is there anyway to do some sort of FIFO so that bits stay on the SSD as long as possible?  Maybe some sort of way I could do it with PowerShell?

[fattipants2016]

If you use file placement rules to make one or more folders exist only on the "SSD" drive, they'll never be moved off since they have nowhere to go.

They'll continuously contribute towards your balancing quota, however, because DrivePool wants the SSD to be empty when it's done balancing.

I used to do this, but life is better now that I created a 2nd partition for these 'temp' files to be moved to.

[fly]

That wouldn't really work for my use case.  I'm hoping for a way to 'age files off' of the SSD.  I have a lot of media and I want the newest media to be on fast disk.

[fly]

Well, it seems this SSD plugin won't work with SnapRAID.  When the drive fills up and the plugin dumps them off to archive, SnapRAID freaks out cause the SSD is now reporting empty.  Anyone got any ideas for workarounds?

[Jaga]

39 minutes ago, fly said:

Well, it seems this SSD plugin won't work with SnapRAID.  When the drive fills up and the plugin dumps them off to archive, SnapRAID freaks out cause the SSD is now reporting empty.  Anyone got any ideas for workarounds?

Don't calculate parity on the SSD drive.  Or..  be prepared to have SnapRAID re-calculate a large amount of it's parity every time your SSD empties out.  It's a volatile volume that you're trying to do parity on, which is naturally going to lead to a lot of activity and re-calcs.

If you really want to do it right, make a RAID 1 mirror with multiple SSDs, then use that as the Drivepool SSD cache, and don't do parity on it with SnapRAID.  You'll still have temporary redundancy in the mirror.

Other options:  speed up your pool by getting faster drives.  Install a fast SSD or NVMe along with Primocache as a L2 caching option against the Pool.  Or manually manage what you keep on the SSD and what gets moved to the pool.

Lots of ways to handle it.  Drivepool wasn't setup to do what you're trying to do with SnapRAID, as SnapRAID likes mostly static archives (or fast disks to work with when there are large numbers of changes).

I'm still unsure why even a 5400 RPM spindle drive can't deliver information fast enough to stream a 4k UHD movie, for example.  My WD Reds @ 5400RPM can easily read/write fast enough to saturate my Gigabit network.  Is there a reason you simply have to have the latest content on a faster SSD?

[fly]

10 minutes ago, Jaga said:

Don't calculate parity on the SSD drive.  Or..  be prepared to have SnapRAID re-calculate a large amount of it's parity every time your SSD empties out.  It's a volatile volume that you're trying to do parity on, which is naturally going to lead to a lot of activity and re-calcs.

If you really want to do it right, make a RAID 1 mirror with multiple SSDs, then use that as the Drivepool SSD cache, and don't do parity on it with SnapRAID.  You'll still have temporary redundancy in the mirror.

Other options:  speed up your pool by getting faster drives.  Install a fast SSD or NVMe along with Primocache as a L2 caching option against the Pool.  Or manually manage what you keep on the SSD and what gets moved to the pool.

Lots of ways to handle it.  Drivepool wasn't setup to do what you're trying to do with SnapRAID, as SnapRAID likes mostly static archives (or fast disks to work with when there are large numbers of changes).

I'm still unsure why even a 5400 RPM spindle drive can't deliver information fast enough to stream a 4k UHD movie, for example.  My WD Reds @ 5400RPM can easily read/write fast enough to saturate my Gigabit network.  Is there a reason you simply have to have the latest content on a faster SSD?

I'd like for newest content to be on SSDs, since it's the content most likely to be accessed.  If I could get FIFO, then it wouldn't be that upsetting to SnapRAID.  What if I just did this 'manually' with PowerShell?  I would assume that DrivePool wouldn't be very happy if I just placed a file on one of the other drives PoolPart folder, but it's worth asking.

I suppose a RAID 1 would work, but somewhat less than ideal since it's basically a 50% storage penalty.

[Jaga]

Drivepool doesn't really care if you manually drop a file/folder inside it's hidden Poolpart folder.  In fact, that's one way to get games to install to the pool correctly as a workaround.  You have to re-calculate the pool space after so Drivepool's stats are current, but that's not a big deal.

So yes, from that perspective you could easily do it with Powershell and scripts.  But you're left with manually doing the balancing yourself when those files come off the SSDs.

Perhaps it's a good feature request for DP's SSD cache too - a FIFO-like system where not all the files come off at once.  Just one checkbox and a slider to determine how "full" you want the SSD to stay.

[fly]

If I create the PowerShell script, and it moves files to different drives, are you saying that DP is alerted to the change instantly and will update it's physical location on the virtual drive?  If so, I'd think it would be relatively trivial for my script to determine the drive with the most space and dump it there!

 

And yes, ideally that would be great.  Probably choose a max used and a min used.  Then once max is hit, dump down to min, based on FIFO.  That would be perfect for my use case.

[Jaga]

3 minutes ago, fly said:

If I create the PowerShell script, and it moves files to different drives, are you saying that DP is alerted to the change instantly and will update it's physical location on the virtual drive?  If so, I'd think it would be relatively trivial for my script to determine the drive with the most space and dump it there!

DP wouldn't know until you told it to measure pool space.  But it wouldn't care either, especially if you (mostly) tried to keep space balanced with the PS scripts.  Just re-measure the pool manually once in a while and it'll be happy.

I suspect the FIFO feature as a request wouldn't be hard to implement.  Can always keep fingers crossed.

[fly]

21 minutes ago, Jaga said:

DP wouldn't know until you told it to measure pool space.  But it wouldn't care either, especially if you (mostly) tried to keep space balanced with the PS scripts.  Just re-measure the pool manually once in a while and it'll be happy.

I suspect the FIFO feature as a request wouldn't be hard to implement.  Can always keep fingers crossed.

If I ran my script nightly, is there a way for it to tell DP to examine the pool again?  Otherwise, if I'm understanding you correctly, it sounds like the files would appear unavailable.

[Jaga]

DP just passes drive operations to the underlying drives.  The files would still be there, and still show in their folders when any app told DP to "go look and see what's there".  Until then, it wouldn't know they were part of the pool, and the pool measurement (for balancing and space used) would appear incorrect.  But you'd still be able to access/stream the files just fine.

I open Scanner and Drivepool daily to check on status and re-measure the pool, and read SnapRAID logs.  Takes all of 3-5 minutes, and gives me peace of mind.  If you had powershell scripts moving files to physical drives nightly, just open Drivepool every now and then and tell it to re-measure.

DP does have a command line utility, but I'm unsure if it's compatible with current version(s), or if it allows re-measuring.

[fly]

1 hour ago, Jaga said:

DP just passes drive operations to the underlying drives.  The files would still be there, and still show in their folders when any app told DP to "go look and see what's there".  Until then, it wouldn't know they were part of the pool, and the pool measurement (for balancing and space used) would appear incorrect.  But you'd still be able to access/stream the files just fine.

I open Scanner and Drivepool daily to check on status and re-measure the pool, and read SnapRAID logs.  Takes all of 3-5 minutes, and gives me peace of mind.  If you had powershell scripts moving files to physical drives nightly, just open Drivepool every now and then and tell it to re-measure.

DP does have a command line utility, but I'm unsure if it's compatible with current version(s), or if it allows re-measuring.

Well then, seems I have a weekend project. 

 

Will report back with script in case anyone else would like to use it.

[Christopher (Drashna)]

On 6/21/2018 at 9:41 AM, Jaga said:

DP does have a command line utility, but I'm unsure if it's compatible with current version(s), or if it allows re-measuring.

"dpcmd remeasure-pool x:"

[fly]

So I'm finally getting around to writing this...

I'm playing with dp-cmd right now and ListPoolParts doesn't seem to work correctly.  Does it need to be in a certain directory to work?

 

	c:\Users\Administrator\Desktop>dp-cmd.exe ListPoolParts
	Error: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'.
	Details:
	System.MissingMethodException: Method not found: 'CoveUtil.OmniStore.Store CoveUtil.OmniStore.Store.Create(Boolean, Boolean, System.IO.DirectoryInfo, Boolean, CustomSerializer)'.
   at DrivePoolCmd.DrivePoolTasks.ListPoolParts()
   at DrivePoolCmd.Program.Main(String[] args)

[Christopher (Drashna)]

it should be "dpcmd" specifically. 

If you downloaded this, then that would be why it's having issues, since it would be a much older version. 

 

When installing the software, it actually installs dpcmd.exe to c:\Windows\system32, so you can access it from anywhere.

And the command should be "dpcmd ListPoolParts X:", where X: is the drive letter of the pool.

[fly]

20 hours ago, Christopher (Drashna) said:

it should be "dpcmd" specifically. 

If you downloaded this, then that would be why it's having issues, since it would be a much older version. 

 

When installing the software, it actually installs dpcmd.exe to c:\Windows\system32, so you can access it from anywhere.

And the command should be "dpcmd ListPoolParts X:", where X: is the drive letter of the pool.

Ah, that's probably it!  The link that @Jaga posted above might need to be updated.

http://wiki.covecube.com/StableBit_DrivePool_Utilities#DrivePool_Command_Utility

 

[Jaga]

Good point - the link and article both need updating.  I didn't realize it auto-installed with the software either.

[Christopher (Drashna)]

I've updated the article to reflect this, so it shouldn't be an issue in the future.

 

 

[fly]

On 6/21/2018 at 11:32 AM, Jaga said:

Perhaps it's a good feature request for DP's SSD cache too - a FIFO-like system where not all the files come off at once.  Just one checkbox and a slider to determine how "full" you want the SSD to stay.

This is turning out much more difficult than anticipated with my limited PowerShell skillset.  How would I go about putting in a feature request?

[Christopher (Drashna)]

Mostly just ask.  

 

[fly]

Pretty please? 

Just to reiterate my use case: My media server has quite a few users.  As new media comes in, it's likely to have a lot of viewers and therefore a lot of I/O.  As files get older, fewer and fewer people access them.  So I'd love to have an SSD landing zone for all new files, which then get archived off to platters as space requires.  It would be fantastic if I could say, once the SSD drive(s) reaches 80% full, archive files down to 50% using FIFO.