Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

SSD Optimizer Balancing Plugin

plugin balancing drivepool

  • Please log in to reply
42 replies to this topic

#21 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 07 February 2016 - 11:37 PM

You're very welcome. 

 

Ya, I figured as much while writing my message and thinking about it. But needing 2 SSDs in order to be able to use the Landing Zone is asking a lot; they're not cheap, and motherboard SATA connectors are at a premium too. So far, without having tried either, I prefer DrivePool over DriveBender simply because support is much better here, but DB does not need 2 SSDs for its Landing Zone...

 

Well, they have dropped in price recently, but yeah, the SATA connectors are a premium still (even if you get controller cards, they can be very expensive, very quickly).

 

And as I said, if you disable realtime duplication, that should "fix" the issue, although with the mentioned caveats. Not idea, but it works.

 

Yes, I was going to get there, and ask about locks and duplication at some point, in another thread. So you are saying that Real Time Duplication hooks low enough on the storage stack that you can duplicate even while the file is locked?

 

Yup, Specifically, with Real Time duplication, any time you write or modify data on the pool, the writes are done to all copies of the file in parallel.  

Essentially, the driver works as a proxy for the file system (this is a drastic oversimplification), and with real time duplication enabled, any file system commands (creating/modifying/etc) are sent to all destination disks at the same time. So locking will not affect the software's ability to write the data.

 

Alex (the developer) has a good post that goes over this feature (from the alpha version years ago), that may be worth taking a read:

http://blog.covecube...vepool-beta-m4/

 

 

And if you disable real time duplication, then this occurs at 2AM normally. However, this can be changed by creating the advanced config file (a "default" version will always be present after installing or updating).

http://wiki.covecube...vanced_Settings

Set the "FileDuplication_DuplicateTime" to a valid time, and this is when it will run a duplication pass. 

Also, changing the duplication status should trigger a duplication pass, and the pool condition indicator (at the bottom) should allow you to re-duplicate manually. 

 

 

 

Yes of course, if you always fill one disk before the others, content will tend to be grouped by folder, but I was hoping that "Folder Balancing" (for lack of a better term) had been added to DP (it was mentionned in another old thread in this forum). Ideally I'd want to fill disks using the default settings (most empty disk first) but grouped by folder. To repeat the example the other guy used, I'd want all 10 tracks of a music CD on a single physical disk (which 10 files are all in one folder) just to make things simpler when accessing the disks when the pool is offline, while still filling-up disks most-empty-first...

 

Well, the Ordered File Placement will attempt to do so, but no, it's not guaranteed to do so. 

 

I believe there is a feature request pending for this already (a balancer that will specifically keep the contents of folders together), and once StableBit CloudDrive is released, we (Alex) should at least get to it, whether or not it's implemented. 

 

Though, as I mentioned, the "File Placement Rules" will allow you to do this, albeit completely manually. (though, you can also do neat things like adding a "\Music\*\*.mp3" to force the mp3 files onto a specific disk, while leaving other files to go anywhere... or add another rule for different extensions).  But it sounds like it's not quite what you're looking for, though.


  • Ginoliggime likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#22 B00ze

B00ze

    Member

  • Members
  • PipPip
  • 11 posts

Posted 09 February 2016 - 01:19 AM

Good day.

 

> And as I said, if you disable realtime duplication, that should "fix" the issue,

> although with the mentioned caveats. Not ideal, but it works.

 

I noted it, thanks! I will probably not duplicate the entire pool anyway, only some folders, so it might be good enough for me to use a single SSD and allow real time duplication, especially if I add a parity disk with SnapRAID or something like that. How often does the SSD Optimizer empty the "Landing Zone" SSDs? Only when the "Fill Threshold" is reached?

 

> Alex (the developer) has a good post that goes over this feature, that

> may be worth taking a read: http://blog.covecube...vepool-beta-m4/

 

Good read! I learned that unlike DB, DP doesn't duplicate the entire folder structure on all disks :-) And that it will hash files when there is a mismatched time. And that it doesn't consider 1 part the "original" and 1 part the "copy" but that all parts are equal. And that +++ it handles other programs modifyng files that it is currently duplicating with no problems. This is all very good.

 

> I believe there is a feature request pending for this already (a balancer that will specifically

> keep the contents of folders together), and once StableBit CloudDrive is released, we (Alex)

> should at least get to it, whether or not it's implemented. 

 

I went to the DEV Wiki and started clicking on tickets, but they are all "private" or something, there is no description on most...

 

> Though, as I mentioned, the "File Placement Rules" will allow you to do this,

> albeit completely manually. (though, you can also do neat things like adding

> a "\Music\*\*.mp3" to force the mp3 files onto a specific disk, while leaving other

> files to go anywhere... or add another rule for different extensions).  But it

> sounds like it's not quite what you're looking for, though.

 

Not quite. The file placement rules are too much for me for now (but the fact that they exist places this product way up in the "Wow so cool" category). I'm not really interested in placing some files on a specific disk (but I might be one day who knows), just in simplifying management when the pool is down. There is no rush; placing files all over the place has some performance advantages anyway.

 

Thank you for the excellent feedback you provide.

Best Regards,



#23 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 09 February 2016 - 03:06 AM

I noted it, thanks! I will probably not duplicate the entire pool anyway, only some folders, so it might be good enough for me to use a single SSD and allow real time duplication, especially if I add a parity disk with SnapRAID or something like that. How often does the SSD Optimizer empty the "Landing Zone" SSDs? Only when the "Fill Threshold" is reached?

 

As for when balancing happens, that depends entirely on the balancing settings. 

 

Specifically, there are three options for frequency: 

  • Do not balance automatically
    • (which requires manual balancing, in most cases)
  • Balance Immediately
    • As soon as the pool condition ratio gets below the specified thresholds, it will start balancing.
    • This can be limited to a specific frequency, in hours, so it's not always moving data around. 
  • Balance every day at ....
    • Will only run once a day, at 1AM by default. 

 

As for the automatic balancing, there is a slider for the ratio. If you want to aggressively move data out of the drive, move this to "100%", otherwise, it won't aggressively move data out, until the drive gets fuller. 

There is also an option to specify "or if at least this much data needs to be moved" option, which may be preferable for the SSD Optimizer.

 

 

This and more is covered in the manual:

http://stablebit.com...ancing Settings

With images, so it may be worth taking a look. 

 

 

 

 

Good read! I learned that unlike DB, DP doesn't duplicate the entire folder structure on all disks :-) And that it will hash files when there is a mismatched time. And that it doesn't consider 1 part the "original" and 1 part the "copy" but that all parts are equal. And that +++ it handles other programs modifyng files that it is currently duplicating with no problems. This is all very good.

 

No, it doesn't. But it may end up with a large majority of it on the drives, over time. But this really depends on how things get balanced. 

 

And yes, specifically, when a file is accessed or a duplication pass occurs (can be triggered by remeasuring the pool, taking a disk offline and reconnecting, or changing the duplication settings), it checks to see if the timestamp matches. If it doesn't, it checks the hash of both files and compares them. If these do match, it updates the timestamp, otherwise it flags the file in the UI as a "mismatched" file and prompts you for resolution. (this is a section that we definitely need to improve, as it's pretty basic right now). 

 

And yeah, there is no distinction between original and copy. Both files are "equal", as there really isn't a reason to differentiate here, and it helps with the Read Striping feature. 

 

 

 

I went to the DEV Wiki and started clicking on tickets, but they are all "private" or something, there is no description on most...

 

This depends on the specific issues. Some don't have public posts on them. We do try to include public information on most of them, even if it's brief.  And some of the issues have a lot of internal info that we don't wish public at this moment (meaning they may be related to potential features and/or products that are only in the planning stage).

 

However, if there are a lot with no description, I'll see about going through and updating some of these. 

 

 

 

Not quite. The file placement rules are too much for me for now (but the fact that they exist places this product way up in the "Wow so cool" category). I'm not really interested in placing some files on a specific disk (but I might be one day who knows), just in simplifying management when the pool is down. There is no rush; placing files all over the place has some performance advantages anyway.

 

Well, we hope that the pool is never down!

 

And as I said, the Ordered File Placement balancer does attempt to do this, but that's not it's main goal.

I've mentioned this to Alex (as I don't remember and can't find a specific request for this), as specifically a balancer that keeps a specific folder on the same disk. 

 

If you have programming experience, we do support 3rd party balancing plugins, and source  to create your own:

http://wiki.covecube...lancing_Plugins

I post this mostly for reference, and if you wanted to take a look at. There is no obligation or anything at all here. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#24 B00ze

B00ze

    Member

  • Members
  • PipPip
  • 11 posts

Posted 10 February 2016 - 01:19 AM

Good day.

 

> Well, we hope that the pool is never down!

 

Hahaha, well I multi-boot and also boot in WinPE regularly, the pool *will* be down sometimes ;-)

 

> And as I said, the Ordered File Placement balancer does attempt to do this,

> but that's not it's main goal. I've mentioned this to Alex (as I don't remember

> and can't find a specific request for this), as specifically a balancer that keeps

> a specific folder on the same disk.

 

Careful, not "a specific folder" but *any* folder, e.g. all files (but not sub-folders) inside any folder are kept on the same disk. The best example of the usefulness of this came from that other guy who talked about this in another thread: 1 Folder per Music Album, perhaps with a sub-folder for the lyrics, and all the tracks inside the folder on the same disk, don't care which, and all the lyrics on the same disk, don't care which.

 

Thank you.

Best Regards,



#25 Erik Wright

Erik Wright

    Newbie

  • Members
  • Pip
  • 7 posts
  • LocationNanaimo, BC

Posted 05 July 2016 - 12:07 AM

To build upon @B00ze's question, how would one use 2 SSD's and 1 HDD to maintain the best possible write speeds with 3x file duplication enabled?

 

Specifically, I have dual 500GB Crucial MX200 SSD's and a single 500GB Hitachi Deskstar P7K500 HDD in my system, configured as a single pool.

I've added the 500GB HDD (hooked up over eSata) to keep a 3rd external copy of my data that I can grab and go in case of fire or other emergency.

 

Before adding the HDD to the pool, I consistently got around 500MB/s write and 1100MB/s read speeds (with read striping enabled) on the pool.

Even with configuring the SSD optimizer, I cannot obtain fast write speeds without disabling 3x duplication on the pool, or disabling real-time duplication.

I understand this is by design, or I've possibly configured the pool settings incorrectly, however...

 

Is there a way to tell DrivePool to write all the initial data ONLY to the SSD's, and then write the 3rd copy of the data to the slower HDD later?

Or should I just remove the HDD from the pool, and use some other backup software to create weekly backups from the pool to the HDD separately?



#26 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 05 July 2016 - 08:14 PM

Is there a way to tell DrivePool to write all the initial data ONLY to the SSD's, and then write the 3rd copy of the data to the slower HDD later?

Or should I just remove the HDD from the pool, and use some other backup software to create weekly backups from the pool to the HDD separately?

 

No. There isn't. 

 

Real Time duplication writes to ALL destination drives in parallel. Period.  

You can disable Real Time duplication, but that has other issues (and we don't recommend it).  This writes to ONLY one disk, and then duplicates later.  

 

The "best" solution may be to write to a folder that is 2x duplicated, and then move to a 3x duplicated folder. That should "solve" the problem without buying a 3rd SSD. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#27 vertigo235

vertigo235

    Member

  • Members
  • PipPip
  • 18 posts

Posted 26 August 2016 - 02:41 PM

Maybe I don't have it set up properly, but it appears that the SSD optimizer is completly ifnoring my file placement options.

 

I want a certain folder to always be on the SSD so I unchecked my Archive drive in File placement, but the optimizer still moves all the files over to the archive drive.

 

Is it supposed to ignore my file placement earmarks?

 

------

 

hmm, well that's weird, it moved everything off on the last balance, but I stepped through the settings screens and clicked saved again.

 

Now it appears to be moving it back, very odd. 



#28 vertigo235

vertigo235

    Member

  • Members
  • PipPip
  • 18 posts

Posted 26 August 2016 - 02:49 PM

---------------------

 

OK so it re balanced like it should have, moving everything I pinned to the SSD back.  Then once done at the bottom it says that the balance was not "optimal" I clicked the re-balance and it appears to be moving my pinned items back off the SSD Again. 

 

I assume it would continue this vicious cycle over and over?



#29 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 26 August 2016 - 04:14 PM

Maybe I don't have it set up properly, but it appears that the SSD optimizer is completly ifnoring my file placement options.

 

I want a certain folder to always be on the SSD so I unchecked my Archive drive in File placement, but the optimizer still moves all the files over to the archive drive.

 

Is it supposed to ignore my file placement earmarks?

 

------

 

hmm, well that's weird, it moved everything off on the last balance, but I stepped through the settings screens and clicked saved again.

 

Now it appears to be moving it back, very odd. 

 

Did you check the notes? 

 

 

* If you've created file placement rules that are attempting to keep files on drives that are designated as SSDs,

  then you should disable the "Unless a drive is being emptied" option, under the "File placement settings" category, 
  on the "General" tab. Otherwise, your File Placement rules will not be respected by this plugin (because it is 
  emptying the SSD drives).

 

Make sure that this setting is unchecked, and it should then behave as expected. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#30 vertigo235

vertigo235

    Member

  • Members
  • PipPip
  • 18 posts

Posted 26 August 2016 - 04:39 PM

I was reading about that but, another oddity I unchecked the SSD optimizer altogether and it still did it.

It seems that it is marking my SSD to have 0 unduplicated files (the blue indicator is all the way to the left) and maybe that is why it keeps clearing it out?

#31 vertigo235

vertigo235

    Member

  • Members
  • PipPip
  • 18 posts

Posted 26 August 2016 - 04:48 PM

OK so that did correct the behavior, but I guess I lose some functionality if there is a drive failure etc.

Can this be a setting at the plugin level instead?

#32 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 31 August 2016 - 03:51 AM

If you mean with StableBit Scanner, I believe that you should be fine, regardless.  It will still respect the real time placement limits created by the StableBit Scanner balancer. 

 

If you want to test out this specific scenario, you can use the "simulate" section in StableBit Scanner. Enable the simulated unreadable sectors option, and let it rescan.  That should trigger the specific behavior (once Scanner flags the disk). 

 

http://stablebit.com...ection=Simulate


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#33 Nikuk

Nikuk

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 19 December 2016 - 02:32 PM

To confirm, and expand upon some thoughts in here:
I had a passing thought that I could split a 250gb ssd into 3 partitioned drive letters as a landing zone. This would address any files that use 3x dupe. I admit that this would be greatly vulnerable to a single point of failure and so is not the best practice at all.
The problem is that I'd be splitting that specific sata port's bandwidth in 3 as well.... so instead of 600 mbps I'd probably be back down to 200 mbps after all. Correct?

Also, if I'm primarily writing to the pool via network transfer I wouldn't necessarily see much advantage anyways..? It's on a gigabit LAN but some devices still use 100mb nic's.

#34 Spider99

Spider99

    Advanced Member

  • Members
  • PipPipPip
  • 442 posts
  • LocationWarrington, UK

Posted 19 December 2016 - 05:03 PM

it should work but you will get less then 200mb/s write to the "other" two drives i would suspect - depends on the size of the files you are copying as you will swamp the cache on the ssd and speeds will drop a fair bit to the "native speed of the drive"/3


Tim

---------

DrivePool & Scanner on 2012R2 Essentials Server Gigabyte Z87x OC Force, i7-4790S, 32 GB, LSI 9211, HP SAS Expander, 27 Storage disks, 6 SSD Cache, ~115 TB Storage


#35 Nikuk

Nikuk

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 19 December 2016 - 05:44 PM

Sorry, let me expand:
The os drive is an ssd. Then I have a pool of 6 6tb HDDs that would be the archival drives. This pool reports ~200 mbps in crystaldiskinfo as expected.
I am considering using an ssd setup as a landing zone with ssd optimizer plug in.
I would need 3 ssd drives to attain ssd r/w speeds and work with 3x dupe, correct?

Then, splitting an ssd into 3 drive letter partitions would defeat the ssd transfer speed goal, correct?

#36 gj80

gj80

    Advanced Member

  • Members
  • PipPipPip
  • 42 posts

Posted 20 December 2016 - 12:13 AM

Splitting an SSD into 3 partitions for feeder-purposes would cut the speed of the writes 3 ways, but if you're on a gigabit lan, that only needs to be ~125MB/s. Good SSDs (samsung 850s, etc) should probably handle 375MB/s sequential write without any issue. I haven't personally tried this, though...just my $0.02

 

Not sure about your use case, but personally I plan to just use the plugin without realtime duplication so I only need one feeder disk without needing to partition anything. The slight risk of losing data I uploaded that same day is something I can live with personally, and I don't have in-use data that I work with on my DP volume which is ever locked up by a program.

 

Alternately...if drive slots are the issue... I know this seems weird, but why not buy one or two of these:

 

https://www.amazon.c... ssd slot mount

 

Then just mount your SSDs inside the chassis and plug them into your onboard SATA ports. That's what I do for all my OS drives. If you buy two, you could stick your OS SSD in one, and then still be able to mount the 3 dedicated feeder SSDs, and you'd actually come out with an additional hotswap bay clear.



#37 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 30 December 2016 - 11:01 PM

To confirm, and expand upon some thoughts in here:
I had a passing thought that I could split a 250gb ssd into 3 partitioned drive letters as a landing zone. This would address any files that use 3x dupe. I admit that this would be greatly vulnerable to a single point of failure and so is not the best practice at all.
The problem is that I'd be splitting that specific sata port's bandwidth in 3 as well.... so instead of 600 mbps I'd probably be back down to 200 mbps after all. Correct?

Also, if I'm primarily writing to the pool via network transfer I wouldn't necessarily see much advantage anyways..? It's on a gigabit LAN but some devices still use 100mb nic's.

 

 

No, this would not work.. StableBit DrivePool does detect what physical drive that the volumes are located on, and will actively avoid placing the duplicates on the same physical disk. 

 

(this is also part of why we don't support Dynamic Disks, as this detection gets INCREDIBLY complex when allowing Dynamic Disks). 

 

You'd need to use three different SSD drives to accomplish this. 


Alternately...if drive slots are the issue... I know this seems weird, but why not buy one or two of these:

 

https://www.amazon.c... ssd slot mount

 

You could also get a few M.2 cards, and a PCI-e  adapter card. 

 

This would use up a lot less space, wouldn't require SATA or power hookups, and allow you to install multiple cards, side by side, getting PLENTY of drives in there. 


  • gj80 likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#38 JasonC

JasonC

    Advanced Member

  • Members
  • PipPipPip
  • 47 posts

Posted 03 February 2017 - 02:23 AM

You might want to mention in the release notes that the plugin cannot be installed on the beta versions of DrivePool. At least that seems to be the case for DrivePool 2.2.0.738.



#39 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 03 February 2017 - 02:32 AM

You might want to mention in the release notes that the plugin cannot be installed on the beta versions of DrivePool. At least that seems to be the case for DrivePool 2.2.0.738.

 

This can happen if the balancer is already installed.  The WIndows Installer service gives a "bad" error code that is misleading. 

 

Check DrivePool to verify if the balancer is present or not. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#40 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 03 February 2017 - 12:32 PM

Hey Drashna.

 

Ive been using Drivepool since back in the WHS/WGS days.

I dont come around here nowadays, simply cos it just works and never really needed much help in receny years.

 

One thing though that I have meant to clarify with the SSD Optimizer (The only balancer I need) I get the settings in the first imageSxXWQFI.png‚Äč

F3YooKy.png

But have never really been sure whether the SSD's should be ticked in the second one here in the file placement options.

I have had them ticked in the past. But even when they aren't it still seems to use the SSD's for cache. (or maybe not)

Just curious if those SSD boxes should be ticked?

 

EDIT... Never mind... Just noticed stuff writing directly to the archive drives with those SSD boxes unticked. :-)







Also tagged with one or more of these keywords: plugin, balancing, drivepool

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users