SSD Optimizer / File Placement Interoperability Issues

[fattipants2016]

I'm not sure if this belongs in General or Nuts and Bolts but here goes.

Basic scenario is I have 4 "drives."

1.) Operating system SSD (not part of the pool.)

2.) SSD landing zone (SSD-optimizer places all new files here for speed.)

3.) HDD 'scratch disk' (folder placement rules place my "temp folder here" when the SSD is emptied in order to hide it from SnapRAID)

4.) Archival drives (folder placement rules migrate my "archival" folders here from either the SSD or scratch disk. This is what SnapRAID sees.)

The issue I'm having is that when balancing occurs, files / folders that are being moved from #3 to #4 are being copied BACK to the #2 first. This creates two issues.

1.) #3 and #4 are equally slow, so copying the files to #2 first means balancing takes twice as long as if they were copied from #3 to #4 directly.

2.) The volume of files being moved from #3 to #4 often exceeds the capacity of #2, so balancing has to be run multiple times for everything to migrate successfully.

I'm not sure if there's something amiss with my placement / balancing rules, this is the intended behavior, or if this is just a scenario that's never come up.

- I have 'file placement rules respect real-time file placement limits' unchecked.

- I have 'balancing plug-ins respect file placement rules' checked.

- I have 'unless drive being emptied' unchecked.

Let me know if I need to clarify anything, I've been playing with it for so long I'm having trouble thinking straight.

Thanks in advance.

 

 

[Christopher (Drashna)]

General it better, Nuts and bolts is more for us to post "how this works" type posts. 

And could you post screenshots of your config? 
Either here, in a PM or at https://stablebit.com/contact/ ? 

And sounds like it's a misconfiguration or misunderstand somewhere. 

[fattipants2016]

Sorry about that, I got side-tracked from being side-tracked by this for a few days.

I verified that this issue is in fact still happening. I moved ~3GB of files from folders on "V:\" to folders on "W:\" and they were in fact moved to "C:\" instead.

Only forcing a second balancing pass finally moved the files from "C:\" to "W:\"

And yes, I realize incorporating "C:\" isn't best practices 

 

[fattipants2016]

So while I'm waiting to hear back, I tried a bunch of configuration combinations to try and resolve the issue.

- disabling "file placement rules respect real-time..." makes new files skip the SSD entirely

- disabling "balancing plug-ins respect file placement rules" makes files go everywhere (expected)

- no combination of these options forces the SSD to ever be used if it's removed from all placement rules

I've also found out that if more data is moved between archive drives than my SSD can hold (250GB) the entire balancing pass gives up, and throws the (paraphrasing) "xxxGB could not be moved ... no suitable destination could be found."

If this is a problem with my settings (pictures above) please let me know!

[fattipants2016]

And to make sure we're on the same page, a much more succinct explanation of the problem (with the visual aids above) goes like this:

1.) New files are copied to drive "Y",  but they're captured by drive "C" (the SSD)' = GOOD

2.) On the next balancing pass, they're moved to their original destination, drive "Y" = GOOD

3.) Same file is moved to a folder on drive "X", but remain on drive "Y" until balancing occurs = GOOD

4.) On the next balancing pass, the files are moved back to drive "C" (the SSD) and stay there = BAD

5.) On the next balancing pass, the files are finally moved from drive "C" to their intended destination, "X" = INDIFFERENT

[Chris Downs]

You mentioned SnapRAID - could you post how that is set up as it might help? This seems like an overly complex arrangement on the face of it... what is your end goal for the setup?

A Drivepool that has an SSD landing drive is clear, but what is SnapRAID doing that you need to hide files from it? 

[fattipants2016]

There's actually several compounding issues. 

- My mechanical hard drives are all in a shared external enclosure which lacks individual-drive power control. I'd like to prevent spinning it up as much as possible.

- Any time a file is removed / changed / added to SnapRAID, data of equal size to the changes is vulnerable on each of the member disks. I'd like updates to happen on a schedule.

- Unlike a true RAID-5 which uses sector-by-sector (file system level) parity data, SnapRAID's file-level parity data actually grows in size when files are changed or replaced. You could potentially end up with a parity (back-up) drive that's completely full, and large chunks of empty unusable space on your data disks.

The SSD addresses issue #1 by accepting new files throughout the day without spinning up the enclosure.

The scratch disk holds files older than a day, until I've decided whether or not they're truly worth committing to the archive, which solves #3.

Together they solve #2, but they're both equally important and serve different problems.

 

[fattipants2016]

Well... I found a stop-gap solution in simply making my landing zone & scratch disk one in the same. This way files only ever move in one direction.

This means I cannot take advantage of my SSD, though, and all files must migrate through my relatively slow 2.5" mechanical scratch disk.

If this is a bug that gets fixed in the future, or there truly is something wrong with my settings, I'd really love it if someone let me know.

[B00ze]

Hmmm, interesting issue, looks like the SSD-Optimizer is used even during balancing passes, which is kinda unexpected. Thanks for the tip that SnapRAID keeps growing (until you do a re-initialize or whatever the command is.) I plan to use a disk bigger than the biggest disk in my Pool for SnapRAID, but was not planning to try to minimize changes to files; maybe I should...

[fattipants2016]

SnapRAID will attempt to fill in the gaps with new files, the parity file only grows by the random unfilled space left over. 

It's trivial per-file, but creating parity data from scratch takes me a full day so I'm just trying to be mindful.

If you ever do run out of space for parity data, you can have it span to another disk, or have SR generate a report about which files are creating the most overhead.

I'm mostly being anal retentive about a lot of this stuff up front, so I can (before long, I hope) forget about it, and let it do it's thing.