Jump to content
Covecube Inc.
  • 0
nevergrownup

File Placement Rules Questions

Question

The new file placement rules are fantastic and a feature I've been awaiting with much anticipation. I have a few questions on their behavior.

 

If I setup a rule on a path and on a child path and a relative rule, what rule takes precedence? For example:

 

\Media\* (Limit to Drive 1)

\Media\Movies\* (Limit to Drive 2)

*.mkv (Limit to Drive 3)

 

Will the movies directory be placed on Drive 1 because it is matched by the first rule? Or will it end up on Drive 2 because the second rule is more specific? If a movie ends in mkv will it be placed on Drive 3? Is there any way to set the priority of the file matching rules?

 

Wildcards are definitely useful in building a rule. Are there any other matching parameters? Could I do a more robust pattern match like:

 

\Media\TV\[0-9]* (Limit to Drive 1)

\Media\TV\[A-L]* (Limit to Drive 2)

\Media\TV\[M-Z]* (Limit to Drive 3)

 

If not, I'd love to see that added

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

The new file placement rules are fantastic and a feature I've been awaiting with much anticipation. I have a few questions on their behavior.

 

Will the movies directory be placed on Drive 1 because it is matched by the first rule? Or will it end up on Drive 2 because the second rule is more specific? If a movie ends in mkv will it be placed on Drive 3? Is there any way to set the priority of the file matching rules?

 

No, actually the rules are combined right now. So if you have a file \Media\Movies\MyMovie.MKV, that will match all the rules. The system now combines the rules in a restrictive way. In other words you've just told the system that you don't want that file on Drives 1, 2, 3. If you have no other drives in the pool it will continue on as if there are no restrictions.

 

And you've kind of hit on something that I've been thinking of since I published build 503. I've talked to Christopher about this over here and I think that we have a nice solution to this.

 

It will work like this:

  • Rules defined in a folders tab cannot be combined with any other rules. So one folder based rule will apply at all times. In the case of when multiple folder based rules match a path, the rule with the highest priority wins.
  • Folder based rule priority is automatic. Rules with more "\" characters in them get higher priority. In other words, rules on deeper directory structures win.
  • Pattern based rules will have an explicit priority that you define by moving the rule up or down in the list, kind of like we already have with the balancers. So if you place your *.MKV rule above the other folder rules then it will win, otherwise the folder rules will win.

I think this makes sense and I'm still thinking about whether there should be a way to combine pattern based rules. Right now I'm leaning towards a no.

 

Expect to see these changes implemented soon. I've already caught a file pattern balancing bug and fixed it in the latest internal BETA. Sometimes it was violating the rules even though the settings told it not to.

Share this post


Link to post
Share on other sites
  • 0

Wildcards are definitely useful in building a rule. Are there any other matching parameters? Could I do a more robust pattern match like:

 

\Media\TV\[0-9]* (Limit to Drive 1)

\Media\TV\[A-L]* (Limit to Drive 2)

\Media\TV\[M-Z]* (Limit to Drive 3)

 

If not, I'd love to see that added

 

As far as regular expression matching, well, first of all it's going to be slower, which is ok for the background balancing pass because that's only done when you change the rules. But we also pass these rules down to the file system driver and it has to evaluate them every time a file is created on the pool. Right now we use a built in function in the Windows kernel that is able to very quickly evaluate whether a path matches a pattern. Moving to regular expressions would mean putting a regular expression parser in the file system driver code. Not an easy task, and it will be slower.

 

It "could" be done, but let's start with these simple ones and see where that goes. This is the first version to support file placement rules, I'd like to flesh out any issues and get them working well.

Share this post


Link to post
Share on other sites
  • 0

No, actually the rules are combined right now. So if you have a file \Media\Movies\MyMovie.MKV, that will match all the rules. The system now combines the rules in a restrictive way. In other words you've just told the system that you don't want that file on Drives 1, 2, 3. If you have no other drives in the pool it will continue on as if there are no restrictions.

 

And you've kind of hit on something that I've been thinking of since I published build 503. I've talked to Christopher about this over here and I think that we have a nice solution to this.

 

It will work like this:

  • Rules defined in a folders tab cannot be combined with any other rules. So one folder based rule will apply at all times. In the case of when multiple folder based rules match a path, the rule with the highest priority wins.
  • Folder based rule priority is automatic. Rules with more "\" characters in them get higher priority. In other words, rules on deeper directory structures win.
  • Pattern based rules will have an explicit priority that you define by moving the rule up or down in the list, kind of like we already have with the balancers. So if you place your *.MKV rule above the other folder rules then it will win, otherwise the folder rules will win.

I think this makes sense and I'm still thinking about whether there should be a way to combine pattern based rules. Right now I'm leaning towards a no.

 

Expect to see these changes implemented soon. I've already caught a file pattern balancing bug and fixed it in the latest internal BETA. Sometimes it was violating the rules even though the settings told it not to.

 

That sounds like an excellent way to handle this on all counts

Share this post


Link to post
Share on other sites
  • 0

As far as regular expression matching, well, first of all it's going to be slower, which is ok for the background balancing pass because that's only done when you change the rules. But we also pass these rules down to the file system driver and it has to evaluate them every time a file is created on the pool. Right now we use a built in function in the Windows kernel that is able to very quickly evaluate whether a path matches a pattern. Moving to regular expressions would mean putting a regular expression parser in the file system driver code. Not an easy task, and it will be slower.

 

It "could" be done, but let's start with these simple ones and see where that goes. This is the first version to support file placement rules, I'd like to flesh out any issues and get them working well.

 

I suspected a full regex parser would a pretty big performance gotcha, but I wasn't sure what you were actually using under the hood to perform your pattern matching. Makes complete sense that the built-in kernel functions would only support the limited pattern substitutions you get under a traditional command prompt.

Share this post


Link to post
Share on other sites
  • 0

Am I missing something? These new file placement rules outlined above are something I'd like to have so I updated plugin to "DrivePool.OrderedFilePlacementPlugin.Setup_1.0.4.5" but can't see any way of specifying a particular folder to a particular disc.

 

Is this because I'm still using DrivePool v1 bcause that's best for WHS 2011 ?

Share this post


Link to post
Share on other sites
  • 0

The rules that we're talking about here are a brand new feature of StableBit DrivePool 2.1.0.503 BETA. They are not available in any previous version.

 

See my blog post for full details: http://blog.covecube.com/2014/04/stablebit-drivepool-2-1-0-503-beta-per-folder-balancing/

 

Download the latest BETA here: http://stablebit.com/DrivePool/Download

Share this post


Link to post
Share on other sites
  • 0

OK thanks Alex - I'd missed that. But it would be really nice to have these features in a WHS 2011 friendly pack. I had a bad experience trying to upgrade Drivepool to v2 in WHS 2011 (outlined in earlier posts) and I'm unlikely to try that again.

Share this post


Link to post
Share on other sites
  • 0

Yes, this has come up before, but it would be impractical for me to implement new features into both versions. Even as it is now, with the existing bug reports and feature requests it's taking much longer than I would like to get release final builds out.

 

But I can offer you my assistance in helping you upgrade to v2 on WHS 2011. If you'd like, and this is entirely up to you, we can set up a remote support appointment and I'll take a look at any issues that you're encountering with the upgrade. Just open up a contact request @ http://stablebit.com/Contact and mention this thread.

Share this post


Link to post
Share on other sites
  • 0

I much appreciate the kind offer but my WHS 2011 is running just fine right now and I want to keep things that way. I've had excellent, rock solid experience changing lots of things with v1 Drivepool and Stablebit Scanner on a WHS 2011 home built server, but everything went to hell in a handcart when I tried to upgrade to v2 Drivepool. The day was only saved with a backup OS image but some folder properties were compromised on the way.  I'm simply not prepared to risk it again until, or unless, changed circumstances force me to.  I guess that the file placement rules are in the "nice to have" category compared to the services which are the raison d'être of the unit. But thanks again.

Share this post


Link to post
Share on other sites
  • 0

That sounds like an excellent way to handle this on all counts

 

I've just finished implementing priorities for file placement rules in an internal BETA (build 510). Available here: http://dl.covecube.com/DrivePoolWindows/beta/download/

 

I've done some preliminary testing and the new priority based rules seem to be working pretty well. I'm going to run this build through the official rounds of testing and release it as a public BETA, if no issues arise.

Share this post


Link to post
Share on other sites
  • 0

 

I much appreciate the kind offer but my WHS 2011 is running just fine right now and I want to keep things that way. I've had excellent, rock solid experience changing lots of things with v1 Drivepool and Stablebit Scanner on a WHS 2011 home built server, but everything went to hell in a handcart when I tried to upgrade to v2 Drivepool. The day was only saved with a backup OS image but some folder properties were compromised on the way.  I'm simply not prepared to risk it again until, or unless, changed circumstances force me to.  I guess that the file placement rules are in the "nice to have" category compared to the services which are the raison d'être of the unit. But thanks again.

 

Well, that's .... wow. Definitely not supposed to happen. :(

 

Do you happen to still have the logs/error reports from around the time that this happened? If so, could you upload them to us, so we can take a look at them?

 

 

And I'm glad to hear that Windows Server Backup saved you when it did go pear shaped.

Share this post


Link to post
Share on other sites
  • 0

I'm afraid it all happened about six months ago, see "Upgrading WHS2011 to 2.x" Oct 24 2013

 

and I no longer have any data on it - just memories of the terror when I realised it was going wrong!

 

The saviour that retrieved the situation was not Windows Server Backup, I've never been very impressed with that, but an Acronis True Image disc image which saved the day, and not for the first time in many years of messing with computers. I'm less than happy not being at the "bleeding edge" of Drivepool developments, especially when I see file placement options passing me by, but I like the integration into the dashboard which v1 gives and, most of all, it just works very well.

Share this post


Link to post
Share on other sites
  • 0

I've just finished implementing priorities for file placement rules in an internal BETA (build 510). Available here: http://dl.covecube.com/DrivePoolWindows/beta/download/

 

I've done some preliminary testing and the new priority based rules seem to be working pretty well. I'm going to run this build through the official rounds of testing and release it as a public BETA, if no issues arise.

 

I just updated to build 510. The order of the rules now looks to be a pretty random jumble and when I manually position them the order reverts to the jumble after I hit save and re-open the list of placement rules. Any ideas?

Share this post


Link to post
Share on other sites
  • 0

Another question on File Placement Rules!!

 

I set all my folders to the specified drives with File Placement Rules. All went well and everything is on the drives that I want it to be on.

However I added a new drive earlier and Drivepool is set to allow anything and everything to go on it. So If my Music is set to be only on Mount point 4 adding another drive (Mount Pont 8) will automatically allow music to be placed on Mount Point 8 after going to all the trouble of making sure that the music was only on Mount Point 4.

Multiply this by about 20 server folders and its a pita.

 

Can I suggest that the default for new drives is that all the Folder Placement checkboxes are unticked and not ticked. I know adding a drive is not an every day event but I still think the default should be to allow nothing on the drive until you allow it. Had I not realised in advance and then hit balance all my neat and tidy music would have again been scattered over 2 drives. :lol:

Share this post


Link to post
Share on other sites
  • 0

Another question on File Placement Rules!!

 

I set all my folders to the specified drives with File Placement Rules. All went well and everything is on the drives that I want it to be on.

However I added a new drive earlier and Drivepool is set to allow anything and everything to go on it. So If my Music is set to be only on Mount point 4 adding another drive (Mount Pont 8) will automatically allow music to be placed on Mount Point 8 after going to all the trouble of making sure that the music was only on Mount Point 4.

Multiply this by about 20 server folders and its a pita.

 

Can I suggest that the default for new drives is that all the Folder Placement checkboxes are unticked and not ticked. I know adding a drive is not an every day event but I still think the default should be to allow nothing on the drive until you allow it. Had I not realised in advance and then hit balance all my neat and tidy music would have again been scattered over 2 drives. :lol:

 

I noticed this as well. It would be nice to be able select multiple placement rules and select/deselect drives for all of those rules at the same time. Something a little like setting access permissions on a group of folders. Drives that differ in a group would be in a semi-selected state unless you tick/untick it. Drives that are common between the rules would appear normally.

 

That way adding a new drive to the pool would be a fairly trivial matter to add to a group of placement rules.

Share this post


Link to post
Share on other sites
  • 0

I'm afraid it all happened about six months ago, see "Upgrading WHS2011 to 2.x" Oct 24 2013

 

and I no longer have any data on it - just memories of the terror when I realised it was going wrong!

 

The saviour that retrieved the situation was not Windows Server Backup, I've never been very impressed with that, but an Acronis True Image disc image which saved the day, and not for the first time in many years of messing with computers. I'm less than happy not being at the "bleeding edge" of Drivepool developments, especially when I see file placement options passing me by, but I like the integration into the dashboard which v1 gives and, most of all, it just works very well.

Well, at least some sort of backup, then. 

Always a good idea to have at least one! 

 

And yeah, "bleeding edge" or "cutting edge" or similar terms are named that way for a reason. Sometimes, you get bit.

 

Another question on File Placement Rules!!

 

I noticed this as well. It would be nice to be able select multiple placement rules and select/deselect drives for all of those rules at the same time. Something a little like setting access permissions on a group of folders. Drives that differ in a group would be in a semi-selected state unless you tick/untick it. Drives that are common between the rules would appear normally.

 

That way adding a new drive to the pool would be a fairly trivial matter to add to a group of placement rules.

(edited quotes for simplicity/length/etc) 

I'm not even going to try and comment on this part. This is all Alex's. :)

I have, however, flagged the thread for him, so he'll respond when he gets a chance.

Share this post


Link to post
Share on other sites
  • 0

I just updated to build 510. The order of the rules now looks to be a pretty random jumble and when I manually position them the order reverts to the jumble after I hit save and re-open the list of placement rules. Any ideas?

 

By default, the rules are now ordered like this (top being highest priority):

  • Manually entered rules are inserted at the top.
  • Folder based rules look for a place to insert themselves starting at the bottom and moving towards the top. Once a folder place rule encounters another folder placement rule with the same path depth or a higher path depth it inserts itself right under that rule. If it doesn't find such a rule then it inserts itself at the top of the topmost folder placement rule.

The system tries to put any existing rules (prior to priority being implemented) into an order that fits the rules above, but I haven't tested that part thoroughly yet.

 

If you're having trouble with your existing rules in build 510, remove all of them, hit save and redefine them.

 

Another question on File Placement Rules!!

 

I set all my folders to the specified drives with File Placement Rules. All went well and everything is on the drives that I want it to be on.

However I added a new drive earlier and Drivepool is set to allow anything and everything to go on it. So If my Music is set to be only on Mount point 4 adding another drive (Mount Pont 8) will automatically allow music to be placed on Mount Point 8 after going to all the trouble of making sure that the music was only on Mount Point 4.

Multiply this by about 20 server folders and its a pita.

 

Can I suggest that the default for new drives is that all the Folder Placement checkboxes are unticked and not ticked. I know adding a drive is not an every day event but I still think the default should be to allow nothing on the drive until you allow it. Had I not realised in advance and then hit balance all my neat and tidy music would have again been scattered over 2 drives. :lol:

 

Ok, I'll add a checkbox for each rule that will say something like "Place files on new drives added to the pool." It will be unchecked by default.

 

I noticed this as well. It would be nice to be able select multiple placement rules and select/deselect drives for all of those rules at the same time. Something a little like setting access permissions on a group of folders. Drives that differ in a group would be in a semi-selected state unless you tick/untick it. Drives that are common between the rules would appear normally.

 

That way adding a new drive to the pool would be a fairly trivial matter to add to a group of placement rules.

 

Multiselect, hmm... Perhaps. I'll see how difficult it would be to adjust the existing code.

Share this post


Link to post
Share on other sites
  • 0

Alright guys, starting with build 511 file placement rules will no longer include added drives by default. There is now a new option to enable that on a rule by rule basis.

 

I've also added the ability to rearrange folder based rules, as long as you don't break folder depth rules. See attached image to illustrate what this looks like now.

 

Download: http://dl.covecube.com/DrivePoolWindows/beta/download/

 

Edit: Multiselect implemented in build 512.

rule_arrangement.png

Share this post


Link to post
Share on other sites
  • 0

Alright guys, starting with build 511 file placement rules will no longer include added drives by default. There is now a new option to enable that on a rule by rule basis.

 

I've also added the ability to rearrange folder based rules, as long as you don't break folder depth rules. See attached image to illustrate what this looks like now.

 

Download: http://dl.covecube.com/DrivePoolWindows/beta/download/

 

Edit: Multiselect implemented in build 512.

 

Fantastic, downloading now.

Share this post


Link to post
Share on other sites
  • 0

Alright guys, starting with build 511 file placement rules will no longer include added drives by default. There is now a new option to enable that on a rule by rule basis.

 

I've also added the ability to rearrange folder based rules, as long as you don't break folder depth rules. See attached image to illustrate what this looks like now.

 

Download: http://dl.covecube.com/DrivePoolWindows/beta/download/

 

Edit: Multiselect implemented in build 512.

 

Re-ordering rules still doesn't seem to work for me. I can drag and drop and sort them exactly how I want, but when I hit save and re-open the rules list they have completely reverted. I reset all settings, de-activated my license, uninstalled, rebooted, deleted C:\ProgramData\StableBit DrivePool, reinstalled, reactivated, and set back up my rules. Same behavior. Not sure what's going on?

Share this post


Link to post
Share on other sites
  • 0

Multiselect works brilliantly. Saved a bunch of time setting up my rules again.

 

Thank you, it was a bit of work but worth it.

 

Re-ordering rules still doesn't seem to work for me. I can drag and drop and sort them exactly how I want, but when I hit save and re-open the rules list they have completely reverted. I reset all settings, de-activated my license, uninstalled, rebooted, deleted C:\ProgramData\StableBit DrivePool, reinstalled, reactivated, and set back up my rules. Same behavior. Not sure what's going on?

 

It could be a bug. Can you take some screen shots before you click save and what you see after?

 

Needless to say, I don't see that over here.

Share this post


Link to post
Share on other sites
  • 0
It could be a bug. Can you take some screen shots before you click save and what you see after?

 

Needless to say, I don't see that over here.

 

Before changes. After changes, but before save. After saving and re-opening.

post-1573-0-94352200-1398744662_thumb.png

post-1573-0-74002900-1398744667_thumb.png

post-1573-0-92924500-1398744672_thumb.png

Share this post


Link to post
Share on other sites
  • 0

Before changes. After changes, but before save. After saving and re-opening.

 

Any idea why it's not preserving the rule order? Is there a log or something that would be useful in diagnosing?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...