Re-installing: new pools, seeding, folder rules, no balancing, beta?

[cryodream]

I had troubles with my first attempt of using DrivePool: file placement rules gone, and Christopher (thank you) advised me to use the newest beta to fix this. So, now I have to reinstall/upgrade the DrivePool program and I would like to recreate my pools from scratch, changing some of the settings. But before hand, I would like to ask some questions, to double-check a couple of things, as I'm gonna be running DrivePool on my main server full of data, which I very much wouldn't like to loose. And running beta version is making me even more uncomfortable.
 
My server:

• Intel Xeon E3-1230 V2
• SUPERMICRO MBD-X9SCM-F-O
• Kingston 32GB KVR16E11K4/32
• LSI 9201-16e
• 4x SGI Rackable SE3016
• ~50 drives atm (~70 eventualy): 4TB, 3TB, 2TB and a couple of SSDs
• Win 7 x64 atm, will re-install win 8.1 x64 next week probably
• Usage: downloading, serving media to HTPCs, backing up other PCs around the household.

What I want to accomplish with DrivePool:

• I will NOT be using any duplication, probably ever. I use syncthing for backups.
• I am gonna make 4-5 pools of various sizes (from 3 to max 16 drives each).
• I don't want any balancing, except StableBit Scanner evacuating the drive, if it shows signs of trouble. StableBit Scanner looks pretty good for now, and probably gonna buy a licence for that one, too.
• I want DrivePool NOT to split folders between the drives, if possible. I explain that one below:

This is an example of how DrivePool split my TV shows on my first attempt of pooling, which is wrong and inconvenient for many reasons. For this example, let's assume my File Placement rules for root folder "TV Shows" is to put files on drives 1 & 2:

Drive 1
    TV Shows (this root folder can be split over drives, obviously)
        Blacklist (this folder should NOT be split over drives, but it is...)
            season 1
                episode 1
                episode 3
                episode 5
                ...
Drive 2
    TV Shows
        Blacklist
            season 1
                episode 2
                episode 4
                episode 6
                ...

How I would like it to be, is to keep files that belong to the same folder on the same drive:

Drive 1
    TV Shows
        Blacklist (now it is not split, as it should be...)
            season 1
                episode 1
                episode 2
                episode 3
                episode 4
                episode 5
                episode 6
                ...
Drive 2
    TV Shows
        Some other show...

Questions:

• The most important question: is there any situation, where by using DrivePool I could loose files? I am talking drivepool bugs (especially using beta version), etc? I tried unRaid years ago, but stopped using, I kept checking in on FlexRaid and now tRaid, but never properly tried them, because of many many people with troubles on the forum. Now finally I chose to try DrivePool, because on the surface it is simple enough and especially, without using duplication, on paper it should be reasonably safe. So I am just double checking...?
• Files are stored as regular NTFS files? ie: if for any reason - machine craps out, windows re-install, stop using DrivePool all together, etc - I simply unhide PoolPart.xxx folders, move my files out to the root of the drives, delete empty PoolPart.xxxx folders and that's that?
• How do I go about upgrading to latest beta? Install beat over current version, or uninstall current version first?
• Very important: I would very much like to do "seeding" of the data to the pool, instead of adding empty drives and copying >100TBs over for no good reason. All the data on all my drives is properly sorted and I would like to keep it that way. Is it actually safe to use "seeding" method, described here: StableBit DrivePool Q4142489 - Covecube - Wiki? I mean is it simply to safeguard people from making some stupid mistakes, or is it actually unsafe and maybe be buggy, or some-such?
• In the wiki (above) in the step 2 I would stop the DrivePool service. But later on, in step 4, it says, that when I move the files to the PoolPart.xxxx folders, I can see the changes and files in the pool right away. How come? Does DrivePool UI work even without the service? I'm just curious, so what does service do?
• The same wiki, step 6 is to start the service, but step 5 points to StableBit DrivePool Q2299585B - Covecube - Wiki, which says, when you "Reset all settings..." the service restarts by itself and starts rechecking. So... I guess, I'll see?
• If I don't wan't any of the files moved after seeding the pool, and I would prefer all the balancers, except Scanner, disabled, would't that mess up my system, if DrivePool starts measuring, and balancing the files automatically, after the service restarts?
• Is it possible to achieve that kind of file placement folder rules, that I have described in the above example? I looked through all the balancing plugins, should I disable most of them?
• StableBit Scanner - always on, right?
• Volume Equalization - disable?
• Drive Usage Limiter - this should not interfere with my organization system, I guess, enabled?
• Prevent Drive Overfill - this one's kinda does the same as the one above, enabled?
• Duplication Space Optimizer - disabled?
• SSD Optimizer - nope.
• Disk Space Equalizer - no, no, no.
• Ordered File Placement - this one sounds like the kind I should install, but I gather I MUST change the settings according to: OrderedFilePlacement/Notes.txt?:
  • UNCHECK "File placement rules respect real-time file placement limits set by the balancing plug-ins."
  • CHECK "Balancing plug-ins respect file placement rules."
  • UNCHECK "Unless the drive is being emptied."
• If I uncheck "Unless the drive is being emptied.", won't it prevent Scanner from moving the files out of dying drives? Or am I misunderstanding what "emptying the drive" means?
• Which balancer actually was responsible of placing episodes from the same folder on separate drives?
• I could not find any 3rd party plugins, are there any? All I could find was those 3 by Covecube...
• If it's not possible to achieve file placement and organization the way I'm looking for, I would be interested in writing a balancer which would ensure files would not be scattered on different drives. You could set "parent" folders in the settings. Parent folders could be spanned across multiple drives, but child folders of those parents, should never be split, except if the drive is full, etc. Would that be possible?
• Actually what takes precedence over what? Balancers or File Placement rules? Because it is a little confusing, the options UI, as it stands now, maybe just because I'm new to this.
• Finally, I am a little worried, that the reason why I'm forced to use Beta is basically, wanting to have multiple pools (as the bug is related to multiple pools). I mean, multiple pools is a core feature, imho, and the fix was probably trivial, looking at the release notes, why not release a new version? It's been, what, 7 months since last release? I'm simply trying to determine, if DrivePool is an active peace of software or is it like for example, unRaid - pretty much one guy calling himself "we/us" and "company", and most importantly, him being gone from the project for years, while everyone's waiting for the ages ago promised new version...?

 

Anyway, if anyone's still reading, this overly long post, thank you for your patience

And thanks in advance for any answers...

[panabodifed]

Ill answer what i know.  

 

The way you want files stored is achievable. But i dont see a reason to create 5 pools. My file placement rules that have never let me down. I dont have any balancer on and File Placement Rules --> Never Allow  files to be placed on any other disks is on.

 

2- Yes, just unhide or move folders.

11 - Drive Usage limiter should be disabled in my opinion, there is no advantage in your setup

12 - Same Above

13 - Same Above

16- Not necessary i think

17-I think you are right. Leave checked.

18-Duplication space optmizer probably. 

20- What you want is possible by default. Balancing > File Placement, look for the TV Show folder and select every disk you want, For the Show 1 folder inside it, select just 1 disk. Be carefull with the except if the drive is full desire, you may end up with the same situation you didnt like, shows across different disks.

21-If Balancing plug-ins respect file placement rules is enabled, rules will come first.

 

If you are moving with caution you can test most things in a virtual environment with multiple disks. You are overthinking my friend, just set like this:

 

*Disable all balancers, except StableBit Scanner. 

*On File Placement you can leave the main folder(TV Shows i.e) as it is by default (splitted on all pooled disks). In Show 1 enable never Allow  files to be placed on any other disks and select the single disk you want it to be stored on.

 

Now show us a photo of this crazy setup with 50 disks 

[Christopher (Drashna)]

Wow, you sure do have a lot of questions!

• In theory, yes. because we're dealing with some low level drivers and access... Also, BSOD's always risk data (by their very nature).

  However, there should be no issues. And we try to take care of any data corruption issues very quickly.

• Yes. The Pool is everything that is in these PoolPart folders. We don't do anything to the files, other than store them there. (we do store some metadata on the pooled disks, but that's the ".covefs" folder and the duplication "tags".

  Additionally, if you need to reinstall or migrate hardware (such as if it fails), then you just need to move the disks over and reinstall DrivePool. As soon as you do, it will see the pooled disks and automatically recreate the pool for you. It may end up with a different drive letter (most likely) but the duplication status and volume label and (*most importantly*) data will remain 100% intact.

• You should be able to upgrade over it. If not, then uninstall it and reinstall it. And the above point also covers this scenario also.
• Absolutely safe. It includes a number of extra steps specifically to prevent balancing while you're adding data, and to make sure that the pool gets properly measured.

  Normally, it updates "on the fly" if you use the pool disk, so seeding bypasses this and causes these stats to get out of sync. However, only the UI and balancing engine need this measured information.

  And I've done this on a dozen systems (some mine, some customers, and some friends). There is absolutely no problem with doing this, because we don't do anything special with the disks.

   

  But as you can see, there are some additional considerations when doing this (those extra steps), which is why we have this outlined. Also, you shouldn't access the "PoolPart" folders normally, except in extreme cases (we consider seeding an extreme case, because ... well 100TBs is a LOT of data!... that would take a couple of weeks!)

• There are two components of StableBit DrivePool. The Driver and the Service.

   

  The Driver is the actual pool. It handles combining the folders into one single disk. So as soon as you move files into the PoolPart folders, the Pool driver will see these files. It handles realtime duplication by writing the files or changes to all the disks required in parallel. It also handles the actual measuring of the pool (the service does kick this off, but the driver handles it).

  But the driver basically concatenates the disks into one "list".

   

  The Service does a number of things. It is the backend for the UI (the UI does very little... basically just communicates with the service). It handles the remote control aspect (in 2.X), the entire balancing engine, and part of the duplication (the driver handles real time duplication, the service maintains/copies/deletes duplication as needed). It also sets the duplication status of folders (which the driver reads).

   

  There are some more stuff that each does, but this is the core of the "bits".

   

  So the duplication and balancing parts are why we have you stop the service. We don't want it moving around files until we're done. And this is the simplest way to accomplish that.

• The reset is to force the service to kick off a remeasure and duplication pass, to ensure the integrity of the pool.

  You could just remeasure the pool, but we recommend resetting it, just in case.

• You can disable all of the balancers, if you want. There is absolutely no issue with doing so, and that definitely won't mess up your Pool at all.

  And you can set up things before or after seeding the pool, because the balancing code won't kick in until the pool is finished measuring.

• There are a couple ways to do this.

  The simplest is to use the Ordered File Placement balancer. This fills up one disk at a time, and tries to keep the content of folders on the same disk.

  The other way is very "micro management"-y. And that is to use the File Placement Rules.

  If you use the File placement rules, there are some caveats with the SSD Optimizer and the Ordered File Placement balancer, but that's about it.

• I'm not sure what you mean by this.

  But StableBit Scanner runs as a background service, so it's always active.

  Also, by default, we query SMART data every couple of minutes, so it's always up to date (this can be throttled though).

  Additionally, when you first start up StableBit Scanner, it should ask what sort of system it is. This configures the default work window for you. And by "work window", this refers to when StableBit Scanner can perform the surface and file system scans. The "server" option only does it at night (when you may not be using it), but the "Desktop" option can run it at any time.

  Also, the scans happen every 30 days, but can be configured to be more or less aggressive.

   

  Also, StableBit Scanner will throttle or stop scans based on disk and controller activity, as well as tempurature.

   

  But if you mean for the balancers, then yes, we always recommend leaving this on (and in the #1 slot).

  You may also want to enable the "move files off in case of SMART warnings" option in the balancer as well (but that's your choice).

• You can turn this one off, especially if you only have one partition per disk.
• If you're not using Duplication, then this isn't really useful for you... except in the case of preparing a disk for removal (uncheck the duplication and unduplication options for a disk and it will evacuate the contents of the disk automatically for you).

  So this balancer doesn't matter as much.

• Prevent Drive Overfill should always remain enabled. It does what it sounds like. Also, getting a disk completely full can cause serious issues with NTFS. A system disk will cause BSODs (or other instability in the system) while a "data disk" will because unstable.

  As for the settings, the "90%/100GBs" slider is for when it gets triggered (if it goes above this, it triggers).

  The "85%/200GB" slider is the target usage that it moves data to when the balancer gets triggered.

  Eg, if the disk gets more than 90% usage, the balancer will cause the disk to be emptied until it hits 85% usage.

   

  We would definitely recommend keeping this enabled in most situations.

• This should also remain enabled, as it helps ensure that ... well duplication is kept optimal.

  However, if you're not using duplication, this won't affect you either way, so you could turn it off.

• The SSD Optimizer balancer is optional and must be installed if you want it.

  If you don't plan on using it at all, then there is no reason to have it installed. But it definitely should be disabled if you're not using it.

• Nope, you definitely don't want the Disk Space Equalizer balancer installed. It's also optional, so must be installed if you do want it.
• If you plan on using the Ordered File Placement balancer (again, optional, so it must be installed), and file placement rules (which honestly, sounds like the best idea for what you want), then YES you need to do this, otherwise, it will cause issues with the file placement/balancing engine.

  (if you want, I can explain this in more detail, but it should be somewhat self explanatory).

• These rules affect how the File Placement rules are handled in regards to the main balancers.

  This one is trickier, because of the Ordered File Placement and StableBit Scanner balancers.

  In this case, I'm not 100% sure, but you'll want to make sure you cehck the "unless the drive is being emptied" option, but make sure the StableBit Scanner Balancer is in the #1 slot.

• None. That's the default strategy, specifically. When adding files to the pool, by default they are placed on the disk with the most free space. So while filling a disk, it may place the files on different physical disks.

  The SSD Optimizer and Ordered File Placement balancers change this strategy.

• As far as I'm aware, there aren't any. For the most part, the built in system handles most scenarios, even if it requires a bit of tweaking.

  However, we do have the API to create 3rd party plugins, in case anyone wants to build one.

• For the most part, the Ordered File Placement balancer should accomplish this. It fills up one disk at a time (instead of the "shotgun" effect of placing files on the disk with the most free space).

  As for more about developing a balancing, let me know if you do plan on doing this, and I'll forward you to Alex (the developer) so you can talk to him more about this.

• The settings in #16 affect the precedence.

  Otherwise, the balancers are listed in order of priority (#1 being the highest priority, and then downward).

   

  I offered to explain in further detail, and is sounds like that's what you want. So here goes:

   

  "File placement rules respect real-time file placement limits set by the balancing plug-ins."

  Enabling this means that files/folders follow the file placement rules ONLY if there is enough room on the disks allowed by the file placement rules.

  Eg, if you have the Ordered File Placement balancer installed and enabled... it sets one or two disks as "allowed" disk via the "Real Time file Placement limits". However, if you have a rule to place a file on a disk that isn't one of these... If this option is enabled, then the files WILL NOT be placed on the correct disk according to the rules.

  And this is why we recommend disabling this option in this scenario.

   

  "Balancing plug-ins respect file placement rules."

  Enabling this means that hte balancers must respect the file placement rules and will not move files off of a disk if there is a rule that says it should be on the disk still. Unless there is another disk that the rule allows the placement to be on.

  Eg, if this is enabled, then the StableBit Scanner or Drive Usage Limiter balancer couldn't force the data off of a disk unless the rule allows for placement on a different disk.

   

  "Balancing plug-ins respect file placement rules. Unless the drive is being emptied." (or just "Unless the drive is being emptied.")

  This modifies the above behavior. If both options are enabled, then when something like StableBit Scanner wants to empty a disk, it will allow it to happen EVEN if the file placement rules wouldn't allow it.

• As for the beta, yes, there is a bug with the balancing settings and multiple pools.

  As for stability, it should be fairly stable. Personally, I update to the newest beta builds once a week and run them. I've never had an issue doing so, but I do understand the hesitancy in doing so.

  Once StableBit CloudDrive is finished (it's getting close), then Alex plans on doing a major overhaul of StableBit DrivePool (we've added some new UI elements that should make troubleshooting a lot easier by providing much better feedback), as well as some tweaks and optimizations based on the CloudDrive code (as it's fairly similar).

   

  Once that's done, we should have a stable release pushed out.

  But the reason for the delay in a release here is that Alex has been focusing mostly on StableBit CloudDrive as it is VERY complicated/complex.

   

  Also, we are very active normally. When Alex doesn't have a huge project that he is trying to get out, the releases are fairly rapid.

  As as for the company, we're two guys. Alex is the owner and developer (sole developer), and I handle most of the customer service and tech support.

  And Alex hates letting stuff sit for too long. He really wants to get stuff out and updated quickly. Though sometimes, development takes a while (and we'd rather take our time and get a bug free version out, than provide rapid releases with tons of bugs).

If you need any clarification here, or have any more questions, don't hesitate to ask!

 

Also, some other info that you may want to know. In StableBit Scanner, in the Disk Settings for each disk, you can specify a "case" and "bay" for the disks. This allows you to sort the disks in StableBit Scanner, and the information shows up in StableBit DrivePool as well (both under the tooltips for the disk, as well as in any email notifications).

So this may make sorting and organizing the disks a lot easier.

 

 

And yes, I agree with panabodifed, it would be awesome to see some pictures of the hardware and/or the software config!

In fact, we have a thread for that:

http://community.covecube.com/index.php?/topic/5-my-rackmount-server/