Jump to content
Covecube Inc.

SSD Optimizer Balancing Plugin


Recommended Posts

  • 1 year later...
  • Replies 83
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

I've just finished coding a new balancing plugin for StableBit DrivePool, it's called the SSD Optimizer. This was actually a feature request, so here you go.   I know that a lot of people use the Ar

By default, it should respect the settings.   You can take a read here at why specifically:  http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Balancing%20Settings     But basically, the f

I have been using it. Seems to be just what I needed. Thanks for this plugin.

Posted Images

Sorry for reopening a very old thread, but @Christopher (Drashna) wondering if you could clarify some terminology for me.

In the options for the SSD Optimizer it has two sliders, ""Fill SSD drives up to:", and "Fill archive drives up to:"  The tooltip that comes up when you hover over says "After this much space is occupied on all of the SSDs, all new files will be placed on the archive drives."

This seems to be pretty clear, but if I turn the % down it still seems it may still be putting files on the SSDs, which is fine actually. What I want is for the SSD to be able to be filled to the brim with stuff that all gets emptied overnight, I'm just not clear on the function of this slider.

My question is:
Does the plugin totally empty the SSD into the archive regardless of this slider's position?
If not, what defines how much gets emptied, the sliders in SSD Optimizer? or the built in "Prevent Drive Overfill" plugin?
If I want to be able to fill the SSD completely with files being ingested into the pool, should I set "Fill SSD drives up to:" to 95-100%?

Appreciate your help
Andy

Link to post
Share on other sites
  • 2 weeks later...

Not Christopher, however the expected behaviour should be that:

  • new files are placed on the "SSD" disks until the limit set by "Fill SSD drives up to" is met, after which new files are placed on the "Archive" disks
  • when balancing is triggered, the SSDs are emptied until the quota set by "Fill archive drives up to" is met or a conflicting rule is encountered.

Q. If I want to be able to fill the SSD completely with files being ingested into the pool, should I set "Fill SSD drives up to:" to 95-100%?

A. Yes*. I would also recommend setting (if it's even enabled at all) the Prevent Drive Overfill balancer to be of lower priority, and set both its fill and empty limits to be equal or greater to the fill limits you set for your Archive disks, to avoid any risk of "backflush" during balancing (I haven't tested if this is even possible, but it seems like a sensible redundancy to set up).

*Caveat: it is generally bad practice to completely fill any drive (in a pool or otherwise) if you can avoid doing so; personally I'd suggest no more than 90% for the "Fill SSD drives up to" limit. Note that if the limit is reached in the middle of writing a file, it will still keep writing that particular file to the SSD anyway (if there's space).

EDIT: Normally I would also recommend setting the Automatic Balancing to "Balance immediately" and picking a GB amount that would ensure the SSD is regularly flushed in the background even while new files are coming in, but you've presumably got a reason for wanting that to only happen at night, so you'd want to select "Balance every day at {insert time at night here}" and set the GB amount to 1.

Tip: if there is a downward-pointing blue triangle above a disk's storage bar in the Pooled Disks section of the DrivePool GUI, this indicates a balancing target that DrivePool will (attempt to) achieve. When the SSD balancer is enabled, a red upward-pointing triangle under each bar indicates each disk's new file placement limit. Hovering your mouse cursor on the triangle should show a tooltip indicating the % limit being applied to that disk. Archive disks should show these red triangles at 0%.

Link to post
Share on other sites
1 hour ago, Shane said:

EDIT: Normally I would also recommend setting the Automatic Balancing to "Balance immediately" and picking a GB amount that would ensure the SSD is regularly flushed in the background even while new files are coming in, but you've presumably got a reason for wanting that to only happen at night, so you'd want to select "Balance every day at {insert time at night here}" and set the GB amount to 1.

When I first setup my SSD on DrivePool, I used your recommended "Balance immediately" and the trigger to "at least 1 GB" needs to be moved. That worked, but I found for my situation a "Balance immediately" and the trigger to "at least 100 GB" needs to be moved worked much better for me. Essentially, I created a very fast 100GB SSD cache file that I use for temp files. My SSD is 248GB, and the 100GB cache has not presented any problems. If I transfer more than 100GB of data, the SSD cache will start to re-balance off to the archive drives and DrivePool also starts to directly write to the archive drives. I have never seen an overflow situation on my SSD and never seen any "backflush."

Link to post
Share on other sites
On 11/25/2020 at 7:09 AM, Andy Plater said:

Does the plugin totally empty the SSD into the archive regardless of this slider's position?

Not Christopher, but think I have some useful info. I have my SSD Optimizer set trigger balancing at 100GB. If I had exactly 100GB of files in the SSD cache and no other files being transferred, the re-balancing would completely empty the 100GB SSD cache into archive drives. In real life, the 100GB threshold usually hits during a transfer of files, so if I had an additional 20GB of data being transferred at that time, the SSD cache would be reduced by 100GB, leaving 20GB still in the cache until the next re-balancing. If I manually tell DrivePool to re-balance with no other files being transferred, it will take my SSD cache down to 0%. I don't think the slider's position makes any difference, it will try to reduce the SSD cache as low as possible in any re-balance.

On 11/25/2020 at 7:09 AM, Andy Plater said:

If not, what defines how much gets emptied, the sliders in SSD Optimizer? or the built in "Prevent Drive Overfill" plugin?

I have the SSD Optimizer on top of all other Balancers. My "Prevent Drive Overfill" is number 5 of 6 on the list. It is my understanding that under the Settings tab, the check mark on "Balance Immediately" and then on the Triggers section "Or if at least this much data needs to be moved: (100GB in my setup)" defines my SSD cache size threshold of 100GB, which triggers the balancing routine, and empties the cache.

On 11/25/2020 at 7:09 AM, Andy Plater said:

If I want to be able to fill the SSD completely with files being ingested into the pool, should I set "Fill SSD drives up to:" to 95-100%?

I have always been told it is bad practice to fill a working drive to full capacity. I agree that 90% on archive pool drives is probably reasonable. I do fill up my offline backup drives to almost 100%. But they are only used for storage and not in a working pool.

However, I have read many times that filling up a working SSD will significantly reduce its life span. It has something to do with the actual media used in SSD's and that they have a limited number of times they can turn on/off. If you keep the data on the SSD down to, say 60%, then the SSD will be able to spread out the data over unused media more evenly, thus extending the service life of the drive. If you fill the SSD over, say 90%, then the data will continually be written to smaller sections of the media and that will prematurely age the drive. I think that is mostly right. My takeaway was that limiting your data usage on an SSD will significantly extend the life of the drive.

But all drives are essentially replaceable, so it's just a matter of understanding that higher fill % on the SSD will mean you will probably have to replace it sooner. Given that the price of SSD's will probably be lower in the future, maybe it is worth it to you to fill your existing drive to 95-100% to achieve your needs now, and then replace the drive when it starts to fail.

I have my SSD to "Fill SSD drive up to: 75%" and "Fill archive drive up to: 90%" This works for me. In any case, I think your archive drive % should be higher than your SSD %, or maybe you might see some "backflush" from the archive drives to the SSD. I don't think you would want that.

Link to post
Share on other sites

 

6 hours ago, gtaus said:

When I first setup my SSD on DrivePool, I used your recommended "Balance immediately" and the trigger to "at least 1 GB" needs to be moved. That worked, but I found for my situation a "Balance immediately" and the trigger to "at least 100 GB" needs to be moved worked much better for me.

Except I didn't recommend "Balance immediately" to be used with "1 GB" trigger? I said to use it with "a GB amount that would ensure the SSD is regularly flushed", while for Andy's particular situation I recommended "Balance every day at {insert time here}" with a "1 GB" trigger since Andy wanted it always able to fill up during the day and always be emptied at night.

6 hours ago, gtaus said:

I have never seen an overflow situation on my SSD and never seen any "backflush."

I was hypothesizing about possible interactions between the SSD Optimizer plugin and the Prevent Drive Overfill plugin during balancing if the latter isn't aware that it shouldn't empty overfill Archive disks back into the SSD disks, as in the notes file for the SSD Optimizer it says "Because this plug-in is very specialized, it should generally be used by itself." So while I'd like to presume it is aware, I'm being cautious.

Link to post
Share on other sites
14 hours ago, Shane said:

Except I didn't recommend "Balance immediately" to be used with "1 GB" trigger?

When I first setup my SSD, I read an old post from 2015 where Christopher (admin) was helping someone setup their new SSD. He stated...

"This depends on the balancing settings. Try setting the balancing to occur immediately, and disable the "but not more often than" option. Also, set the ratio slider on the main tab to "100 %" and setting the "or needs to move this much" to "1GB". This should help it be more aggressive about moving data out. And this should definitely work, as that's exactly what I'm doing on my system, and it's constantly moving data off of the 2x128GB SSDs I have."

I may have conflated Christopher's older post with your recent comments. You clearly said... 

22 hours ago, Shane said:

Normally I would also recommend setting the Automatic Balancing to "Balance immediately" and picking a GB amount that would ensure the SSD is regularly flushed in the background even while new files are coming in,

In my case, I set the GB amount to 100 GB and it works great for me. When I had the GB amount set to 1 GB, DrivePool was constantly re-balancing during even the smallest file transfers. Thanks for the clarification of your comment which is also more consistent with my understanding of the GB amount - set it higher GB amount but still at a point where it is flushed regularly....

Link to post
Share on other sites

Announcements


×
×
  • Create New...