Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

SSD Optimizer Balancing Plugin

plugin balancing drivepool

  • Please log in to reply
40 replies to this topic

#1 Alex

Alex

    Lead Programmer

  • Administrators
  • 241 posts
  • LocationNew York, USA

Posted 09 May 2014 - 12:01 AM

I've just finished coding a new balancing plugin for StableBit DrivePool, it's called the SSD Optimizer. This was actually a feature request, so here you go.

 

I know that a lot of people use the Archive Optimizer plugin with SSD drives and I would like this plugin to replace the Archive Optimizer for that kind of balancing scenario.

 

The idea behind the SSD Optimizer (as it was with the Archive Optimizer) is that if you have one or more SSDs in your pool, they can serve as "landing zones" for new files, and those files will later be automatically migrated to your slower spinning drives. Thus your SSDs would serve as a kind of super fast write buffer for the pool.

 

The new functionality of the SSD Optimizer is that now it's able to fill your Archive disks one at a time, like the Ordered File Placement plugin, but with support for SSD "Feeder" disks.

 

Check out the attached screenshot of what it looks like

 

Notes: http://dl.covecube.c...mizer/Notes.txt

Download: http://stablebit.com/DrivePool/Plugins

 

Edit: Now up on stablebit.com, link updated.

Attached Thumbnails

  • ssd_optimizer.png

  • Christopher (Drashna), CosmicPuppy, postvlad and 2 others like this

#2 gringott

gringott

    Member

  • Members
  • PipPip
  • 27 posts

Posted 28 June 2014 - 03:50 PM

I look forward to trying this one out.



#3 gringott

gringott

    Member

  • Members
  • PipPip
  • 27 posts

Posted 06 July 2014 - 05:53 AM

I have been using it. Seems to be just what I needed. Thanks for this plugin.


  • Alex and Christopher (Drashna) like this

#4 4Frame

4Frame

    Advanced Member

  • Members
  • PipPipPip
  • 35 posts
  • LocationNew York, USA

Posted 03 August 2014 - 08:30 PM

I read on another thread someone was stating that this needs 2 SSD's to work, otherwise it uses 1 SSD + Mechanical for feeding. Is that true? 


Electronic music producer with a whole ton of project files on lots of hard drives. The world of storage is amazing and really important to me. 

http://www.soundcloud.com/4Frame


#5 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 04 August 2014 - 06:51 PM

@4Frame,

 

Yes, that is absolutely correct. If you check the notes link, the 3rd bullet point states this. Specifically:

If you are using duplicated files, then you should specify multiple SDD disks or else the system will fall back 

  to an Archive disk for one or more of the file parts.

  • CosmicPuppy likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#6 Ayala

Ayala

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 19 March 2015 - 05:24 PM

What size SSD would I benefit from? do 120GB SSD's do the job?



#7 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 20 March 2015 - 08:20 PM

That really depends on the size of the data you're adding to the pool, and how much data your adding at any one time.

 

For the most part, 120GB drives should be good.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#8 propergol

propergol

    Advanced Member

  • Members
  • PipPipPip
  • 80 posts
  • LocationFrance

Posted 03 October 2015 - 07:00 PM

I have started testing SSD plugin and there is something I don't anderstand : originaly I was thinking that SSD plugin would write data first to SSD and then copy data to HD in background. But for now I see data are staying on the SSD and are note moved to HD  :wacko:  even with "Fill SSD drives up to : 1%".

SSD Optimizer is at the top of balancers list.



#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 03 October 2015 - 07:37 PM

I have started testing SSD plugin and there is something I don't anderstand : originaly I was thinking that SSD plugin would write data first to SSD and then copy data to HD in background. But for now I see data are staying on the SSD and are note moved to HD  :wacko:  even with "Fill SSD drives up to : 1%".

SSD Optimizer is at the top of balancers list.

This depends on the balancing settings. 

 

Try setting the balancing to occur immediately, and disable the "but not more often than" option. Also, set the ratio slider on the main tab to "100 %" and setting the "or needs to move this much" to "1GB". 

 

This should help it be more aggressive about moving data out. 

And this should definitely work, as that's exactly what I'm doing on my system, and it's constantly moving data off of the 2x128GB SSDs I have.


  • propergol likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 propergol

propergol

    Advanced Member

  • Members
  • PipPipPip
  • 80 posts
  • LocationFrance

Posted 03 October 2015 - 07:57 PM

Thanks a lot : it instantly worked  :)

For now I am still testing SSD pluggin on unduplicated files. I am waiting for a third 240gb SSD to arrived (Sandisk PRO, not the slowest SSD  :ph34r: ...) to start testing with x3 duplication.

From what I can see for now, it would work very well!


  • Christopher (Drashna) likes this

#11 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 04 October 2015 - 05:55 PM

Glad to hear it!

 

And I'll see about getting that added to the documentation. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#12 propergol

propergol

    Advanced Member

  • Members
  • PipPipPip
  • 80 posts
  • LocationFrance

Posted 04 October 2015 - 08:15 PM

As I was testing SSD plugin, copying big movies (30GB), I think to something that I don't know if its possible or not :

now the file is beeing copied first to the SSD then when the entire file is on the SSD, it is then copied to the HD in background.

In order to speed up things, I was wondering if on big files it could be possible to devide the copy task into chunks file=A+B+C....?

This way, copy of part A from the SSD to HD would start as soon as part A is fully copied on the SSD and not only when A+B+C are on the SSD.

I don't know if I am clear enough  :D ...to sum up : something that would parallelize wrhite to SSD cache/ read and copy from SSD CACHE.

 

EDIT : forget to say that the other idea is to use SSD write/read to its max. On fast SSD I think that simultaneous W/R can be arround 500MB/s, not sure, but defenitly much faster that HD read or write.



#13 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 05 October 2015 - 05:19 AM

Are you doing this over the network? 

If so, then the max throughput is 125MB/s (not accounting for overhead).  So you're going to see 110-120MB/s tops, most likely. Adding more tasks will split the bandwidth.

If you're using 10gig networking, then ... this isn't accurate.

 

 

As for multithreaded copies, it depends.  If real time duplication is enable,d all copies of the files are written to all drives in parallel. More threads split the throughput. So, it may not actually help. However, if you're copying from multiple spinning drives to the pool at the same time, then it will definitely benefit.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#14 Gishi

Gishi

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 06 October 2015 - 11:19 AM

is there real time read as well with this plugin? because i do a lot of work editing raw video and some times i have to move video to my 60gb ssd and i have a problem with running out of space.



#15 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 06 October 2015 - 07:25 PM

is there real time read as well with this plugin? because i do a lot of work editing raw video and some times i have to move video to my 60gb ssd and i have a problem with running out of space.

I'm not sure what you mean here.

 

There is the read striping feature which may boost read speeds for you.

 

Aside from that, there is the file placement rules, which you could use to lock certain files or folders to the SSDs to get better read speeds.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#16 propergol

propergol

    Advanced Member

  • Members
  • PipPipPip
  • 80 posts
  • LocationFrance

Posted 23 December 2015 - 10:06 PM

Christopher or Alex, did you had the time to check how single write (x1) is managed by SSD pluggin when multiples SSD are used as cache?

I can confirm that always the same SSD is used if you make a copy to a pool with xxx SSD.

I think you anderstand that this lead to wear on this SSD rather than dispatch the write, one time one SSD 1, an other time on SSD 2 and so on.

I guess a simple random choice will do the trick, but maybe it is more compicated to code, I don't know.



#17 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 30 December 2015 - 06:58 PM

No, we haven't had a chance to really look into it.  

Alex is the one that does this (he's the developer, and i'm the technical support and customer service guy). 

 

Though, I'll bring it up to alex.


  • CosmicPuppy likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#18 B00ze

B00ze

    Member

  • Members
  • PipPip
  • 11 posts

Posted 07 February 2016 - 09:33 PM

Good day.

 

I am reading-up on DriveBender and DrivePool. Whichever solution I enventually pick, I will want the Landing Zone feature, so here I am. I have a question or two regarding the SSD Optimizer.

 

First, this:

 

If you are using duplicated files, then you should specify multiple SDD disks or else the system will fall back to an Archive disk for one or more of the file parts.

 

 

I don't fully understand this. The Landing Zone, to me, should not duplicate files "on the fly" - it's only a Landing Zone. Whatever files are copied to the zone should be duplicated later when they are moved off the zone and into the archive disks. Ok, so the statement above kinda says that files going to the Landing Zone are duplicated on the fly. Question: If a duplicated file (x2) is dumped into the zone, and I have only a single SSD, then one part will go to a disk and one part to the zone. So using a Landing Zone is useless then? The I/O will be slow anyway since one of the parts is going to a disk. Do I understand this correctly? If I am understanding correctly then this severely limits the usefulness of this plugin :-(

 

Then there is this for this pluging: Ordered File Placement ->

This is just for filling up volumes in a specific order? Replacing the other plugin of that name?

 

Finally, do you have per-folder balancing yet? Where files inside a folder are kept together on a single disk (sub-folders can be placed on another disk, what counts is that files inside a folder are grouped together). If you do have it, can it be used together with the SSD Optimizer?

 

Thank you.

Best Regards,

 



#19 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 6,944 posts
  • LocationSan Diego, CA, USA

Posted 07 February 2016 - 09:58 PM

Specifically, the reason for the multiple disk requirement is the Real Time Duplication feature. This feature specifically writes out all copies of the files, to the disks in the pool in parallel.  

 

The reason this is a big feature is that when a program opens a file, it locks it. That means it can't be moved, copied or otherwise altered while the file is open. This includes data on the pool, and definitely affects the ability of our software to duplicate the data. 

And why does this matter? Well, for instance, if you have a database program, and you're storing the database on the pool, the files will ALWAYS be locked. So, StableBit DrivePool would not be able to duplicate the files. 

 

 

However, you absolutely can turn off Real Time Duplication (we don't recommend it), which means you'd only need the one disk. However, this means that the data is only duplicated at a specific time, and once per day. If the files are locked during that time, then the data doesn't get duplicated.  And that's why we don't recommend disabling Real Time Duplication. 

 

Also, the "SSD" drives don't have to be actual SSDs. They can be any sufficiently fast drives. 

 

 

 Question: If a duplicated file (x2) is dumped into the zone, and I have only a single SSD, then one part will go to a disk and one part to the zone. So using a Landing Zone is useless then? The I/O will be slow anyway since one of the parts is going to a disk. Do I understand this correctly? If I am understanding correctly then this severely limits the usefulness of this plugin :-(

 

If you have duplication (x2) enabled, are using the SSD Optimizer, have Real Time Duplication enabled, and only have one "SSD", then yes, it will fall back onto one of the "Archive" drives, and yes, you may see slower throughput because of it.

 

 

 

 

As for Ordered File Placement Balancer Plugin (it doesn't replace any other balancers, but it *is* included in the SSD Optimizer due to technical reasons),  it changes the pool's default "new file placement" strategy, which is to place new files on the disk with the most available free space, which will equalize the disk usage over time.   This balancer fills one (or two, with duplication) disks at a time. 

This balancer will also tend to keep the contents of a specific folder on the same disk (but it's not guaranteed).

 

 

Additionally, there is the "File Placement Rules" which can be used to "lock" specific files or folders to specific disks. Though, this may require more micromanaging of the data than you'd prefer.

You can see details about this feature here:

http://blog.covecube...lder-balancing/

 

And yes, you can use this in conjunction with the SSD Optimizer. However, if you do, you need to disable the "Unless the disk is being emptied" option in the main Balancing settings to get it to keep the files on the SSD drive. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#20 B00ze

B00ze

    Member

  • Members
  • PipPip
  • 11 posts

Posted 07 February 2016 - 10:37 PM

Thanks for the quick response.

 

> Specifically, the reason for the multiple disk requirement is the Real Time Duplication feature.

> This feature specifically writes out all copies of the files, to the disks in the pool in parallel.

 

Ya, I figured as much while writing my message and thinking about it. But needing 2 SSDs in order to be able to use the Landing Zone is asking a lot; they're not cheap, and motherboard SATA connectors are at a premium too. So far, without having tried either, I prefer DrivePool over DriveBender simply because support is much better here, but DB does not need 2 SSDs for its Landing Zone...

 

> The reason this is a big feature is that when a program opens a file, it locks it.

> That means it can't be moved, copied or otherwise altered while the file is open.

> This includes data on the pool, and definitely affects the ability of our software

> to duplicate the data.

 

Yes, I was going to get there, and ask about locks and duplication at some point, in another thread. So you are saying that Real Time Duplication hooks low enough on the storage stack that you can duplicate even while the file is locked?

 

> And why does this matter? Well, for instance, if you have a database program,

> and you're storing the database on the pool, the files will ALWAYS be locked. So,

> StableBit DrivePool would not be able to duplicate the files. However, you absolutely

> can turn off Real Time Duplication (we don't recommend it), which means you'd

> only need the one disk. However, this means that the data is only duplicated at a

> specific time, and once per day. If the files are locked during that time, then the

> data doesn't get duplicated.

 

Hmmm, that might be ok with me; Thanks for mentionning it!

We can, I assume, specify how often the duplicating job runs and when?

 

> As for Ordered File Placement Balancer Plugin (it doesn't replace any other

> balancers, but it *is* included in the SSD Optimizer due to technical reasons),

> it changes the pool's default "new file placement" strategy, which is to place

> new files on the disk with the most available free space... This balancer will

> also tend to keep the contents of a specific folder on the same disk (but it's

> not guaranteed).

 

 

Yes of course, if you always fill one disk before the others, content will tend to be grouped by folder, but I was hoping that "Folder Balancing" (for lack of a better term) had been added to DP (it was mentionned in another old thread in this forum). Ideally I'd want to fill disks using the default settings (most empty disk first) but grouped by folder. To repeat the example the other guy used, I'd want all 10 tracks of a music CD on a single physical disk (which 10 files are all in one folder) just to make things simpler when accessing the disks when the pool is offline, while still filling-up disks most-empty-first...

 

Thank you.

Best Regards,







Also tagged with one or more of these keywords: plugin, balancing, drivepool

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users