All In One Balancing Plugin

[methejuggler]

In an attempt to learn the plugin API, I created a plugin which replicates the functionality of many of the official plugins, but combined into a single plugin. The benefit of this is that you can use functionality from several plugins without them "fighting" each other.

I'm releasing this open source for others to use or learn from here:

https://github.com/cjmanca/AllInOne

Or you can download a precompiled version at:

https://github.com/cjmanca/AllInOne/releases

Here's an example of the settings panel. Notice the 1.5 tb drive is set to not contain duplicated or unduplicated:

 

Balance result with those settings. Again note the 1.5tb is scheduled to move everything off the disk due to the settings above.

[Spider99]

Would be good if you supplied the dll needed or if i missed it install instructions

as most users wont have VS to compile their own copy

[methejuggler]

Take a look at the "Releases" link on the right side of the github page. I'll edit the post with a link to there too so people can find it easier.

[gtaus]

I see you now have 4 options: SSD - Archive - Duplicated - Unduplicated. Would you please explain how the new Duplicated and Unduplicated options work? I understood the current either/or option between SSD and Archive, but I am not sure about your new options. For example, I don't understand why you want a SSD with Duplicated and Unduplicated files. Should not the SSD drive be primarily a temp cache drive?

If I had an older, slower, drive in my pool and maybe it was nearing its end of life, would I only designate it as Archive - Duplicated? That way, if it failed, only duplicated files would be lost and those files could be rebuilt from the original files on the newer drives. 

[methejuggler]

11 hours ago, gtaus said:

I see you now have 4 options: SSD - Archive - Duplicated - Unduplicated. Would you please explain how the new Duplicated and Unduplicated options work?

These are the same options you'd find in the "Drive Usage Limiter" plugin. They limit what data can be placed on each drive. I'm not sure how useful they would be for SSD drives, but for archives they allow you to specify that a certain drive shouldn't have unduplicated data placed on it, for example. Having these checked for SSDs just specifies which type of content can be placed there initially. The content is still moved off onto the archive drives later.

 

11 hours ago, gtaus said:

If I had an older, slower, drive in my pool and maybe it was nearing its end of life, would I only designate it as Archive - Duplicated? That way, if it failed, only duplicated files would be lost and those files could be rebuilt from the original files on the newer drives. 

Exactly. It lets you specify suspect drives as only having duplicated data.

[gtaus]

14 hours ago, methejuggler said:

These are the same options you'd find in the "Drive Usage Limiter" plugin.

Thanks. I have never looked at that plugin. I can see where you just combined all those options in one balancer. Very nice.

14 hours ago, methejuggler said:

Exactly. It lets you specify suspect drives as only having duplicated data.

Great. I just pulled out some old drives that are testing out Health at 54% - still working but suspect and nearing end of life. I would like to use them in DrivePool but only for the duplicate files in case the drive fails. Now I know how to set that up. DrivePool saves me money again....

[methejuggler]

8 minutes ago, gtaus said:

Great. I just pulled out some old drives that are testing out Health at 54% - still working but suspect and nearing end of life. I would like to use them in DrivePool but only for the duplicate files in case the drive fails. Now I know how to set that up. DrivePool saves me money again....

Careful - DrivePool doesn't make any distinction between the two duplicates... they're both considered a "duplicate". So if it puts both duplicates of the same file on two different "bad" drives, and both of those drives go, you lose the file. This is the reason I was looking at drive groups (and ended up solving it with hierarchical pools).

[methejuggler]

Example of my current setup with hierarchical pools: I have my end of life disks all in one pool, and each of the other pools has 2 drives each, purchased at the same time (I placed the purchase dates in the name to tell them apart easier). One of the 8tb seagate drives already has a pending sector on it, so I don't trust it (so no unduplicated content). Will probably RMA that soon. The 14tb drives are solid so far. The new Western Digital 8tb drives are still being scrubbed by the scanner, so I have it set not to place content on them yet until that finishes.

 

Due to some strange glitch with DP while I was re-seeding and moving things to the new hierarchical pool setup, DP deleted a bunch of my duplicates, so it's in the process of re-duplicating, but you can see the balancer would be attempting to move all the unduplicated to the 14tb, nothing on the 8tb WD's, everything off the "SSDs", and then balance the remaining 3 evenly. Note that it takes the "unpooled other" files into account for the even percentage balancing.

[gtaus]

32 minutes ago, methejuggler said:

Careful - DrivePool doesn't make any distinction between the two duplicates... they're both considered a "duplicate". So if it puts both duplicates of the same file on two different "bad" drives, and both of those drives go, you lose the file. This is the reason I was looking at drive groups (and ended up solving it with hierarchical pools).

Ok, after much thought, I think I understand your point. If I have 2 or more suspect drives, and mark them for duplicates only, I could be setting myself up for data loss on my important data because both the original and duplicate files might end up on both of those suspect drives which are more likely to fail then my newer drives. In that case, maybe I would be better off marking the suspect drives for unduplicated files only and restore the files from my offline backup drives in case the suspect drive(s) die.

Specifically, in my case, my original media files are stored on offline backup drives. If DrivePool lost an unduplicated file due to a suspect drive failure, I still have the original file(s) on my backup drives in the closet. And, to limit the risk of loss to only my replaceable media files, I might want to put those suspect/end of life drives into a separate drive group like you have done. Did I get that right? Thanks.

[methejuggler]

8 minutes ago, gtaus said:

In that case, maybe I would be better off marking the suspect drives for unduplicated files only and restore the files from my offline backup drives in case the suspect drive(s) die.

I don't know about that. The solution I went with is to add all suspect drives to a single pool (no duplication on that pool), then add that pool to my main pool, so it acts as a single drive - meaning only a single copy of the duplicated files will ever be placed on any suspect drive.

[gtaus]

3 hours ago, methejuggler said:

I don't know about that. The solution I went with is to add all suspect drives to a single pool (no duplication on that pool), then add that pool to my main pool, so it acts as a single drive - meaning only a single copy of the duplicated files will ever be placed on any suspect drive.

Yes, that's a much better solution. I think I will go that way myself.

Currently I am running some diagnostics and disk repair routines on my older drives that were sitting in the closet unused. For some reason, they pass standard read/write tests, but fail other more in depth diagnostic tests. Appears that bad sectors were reallocated and blocked from use. So maybe that is why the HDD(s) can now pass standard read/write tests, but still fail the more in depth diagnostics? Don't really know. I just call them old, suspect drives that mostly work but might not for much longer. Still, I would hate to throw out a mostly good drive just because a few bad sectors had to be reallocated and blocked from use.

I think your solution may be the best solution to using those suspect drives in DrivePool with no real loss if a drive fails. DrivePool should recognize the loss of a failed drive and then rewrite missing duplicate file(s) to another drive.

[Shane]

Have now been using your plugin for about a fortnight with no issues, in conjunction with the Stablebit Scanner and Prevent Drive Overfill balancers.

Thankyou!

[gtaus]

On 12/23/2020 at 6:27 PM, Shane said:

Have now been using your plugin for about a fortnight with no issues, in conjunction with the Stablebit Scanner and Prevent Drive Overfill balancers.

Thankyou!

@methejuggler or @Shane, Is this the All In One by C. J. Manca plugin which is on the official DrivePool plugin page? If I install the All In One plugin, should I uninstall the other plugins as no longer necessary? Am I correct in thinking that the Stablebit Scanner plugin is still needed?

Also, MS Defender is blocking the All In One plugin download stating "it may harm your device." The other plugins on the page did not have that warning and downloaded without any problems. Any reason why the All In One .msi plugin is being blocked and the other .msi plugins download normally? Of course, I can manually override the warning, but others who have not been following this thread may be hesitant to download the file given the MS Defender warning.

[Shane]

Yes, it's the same plugin. Congrats @methejuggler!

If you're using All In One then you don't need SSD Optimizer, Ordered File Placement, Disk Space Equalizer and Drive Usage Limiter.

Yes, Stablebit Scanner is NOT replaced by All In One.

MS Defender may be blocking the plugin download because it's not familiar with the file, and (at least in my experience) its default reaction to a new executable is to claim it may harm your device, and/or because the downlink link is using http from a https page (I'll let Alex know; EDIT: fixed).

 

[methejuggler]

Shane beat me to it, but yes you still need the Scanner plugin. I'd imagine that it'll stop giving that warning in a week or two when enough people have downloaded it and it has enough "sample size" to know it's ok.

[Shane]

Bug: selecting "Fill up the disks in the order specified below" instead selects "Place files wherever there's room" and if you then select either of the "Equalize by" options then the "Place files wherever there's room" also remains permanently selected.

Workaround: resetting the balancer to its default values and then re-applying preferred values without clicking on "Fill up the disks in the order specified below".

[gtaus]

Thanks for the replies. I downloaded the All In One plugin and reported it as safe to MS. Hope that helps.

[methejuggler]

15 hours ago, Shane said:

Bug: selecting "Fill up the disks in the order specified below" instead selects "Place files wherever there's room" and if you then select either of the "Equalize by" options then the "Place files wherever there's room" also remains permanently selected.

Workaround: resetting the balancer to its default values and then re-applying preferred values without clicking on "Fill up the disks in the order specified below".

Very strange, it didn't used to do that when I was testing, I wonder when that bug crept in. I'll fix it when I get home

[methejuggler]

22 hours ago, Shane said:

Bug: selecting "Fill up the disks in the order specified below" instead selects "Place files wherever there's room" and if you then select either of the "Equalize by" options then the "Place files wherever there's room" also remains permanently selected.

Workaround: resetting the balancer to its default values and then re-applying preferred values without clicking on "Fill up the disks in the order specified below".

I can't replicate this. The buttons appear to be working fine for me. Were there any extra steps that you took to make this happen?

Edit: I found it. Only happens if no drives are set as "SSD". I released a new version to fix this.

[Ultradianguy]

@methejuggler @Shane I had a couple questions/requests regarding this very helpful balancer (these address gaps that were present in the official covecube balancers as well).

Currently, there is no way to do what seems most useful to me - have equal space used for unduplicated data across drives.  That is, if I have 20TB of data , and 4 drives, I'd like 5 TB on each drive, so that if one goes, I minimize the loss of unduplicated files.

Currently, you can do this by % of space used, which does the same thing if the drives are about the same size.  But over time, I've added larger drives as prices have dropped.  So I have a mix of 2TB, 4TB, 6tB , 8TB and now a 20TB drive.

If I use equal % used or equal empty space, I end up with far more data on the larger drives.  This seems like a pretty obvious use case, and would be easily resolved by an option to equalize by actual space used, rather than percent. There is currently no way to equalize by absolute amount of data. 

 

The other issue I'm having was with the covecube balancers. I just switched to All in One so it remains to be seen if it will still be an issue  But I found that when I set the balancers to equalize by percent used for unduplicated (unchecked duplicated), that the balancing ignored that and equalized total space used.  As a result there were very large differences in unduplicated data across drives.  I wasn't sure if the new all in one uses different code or just puts the existing code into a better UI . 

 

Thanks for the work on this!