Drive Balancing Duplicating files?

[BuddyRich]

Ive just started the trial of DrivePool and Im trying to configure it so I can use it with snapraid (vs. your duplication).
 
I think its working but I may have encountered a bug.  I have 4x3TB drives.  Ive made a pool with 3x3TB, leaving 1 for the snapraid parity drive.  So far so good, I went ahead and created three folders in the pool, movies, music and tv.
 
I have the following set under balancing:
 
 
 
And am using the ordered file placement balancer only with these settings:
 
 
 
I set up the following file placement rules:
 
 
 
 
With the same rule on TV, but selecting drive J, I want to leave I to be a spillover for my movies (as I know I will be over 3TB on it).
 
So I copied some movies to the movies folder on the pool and it went to the H drive as expected.
 
Next I copied some tv shows and its showing up on I and J at the same time.  Its like its respecting the balancer plugin and moving it to I: and also the file placement rules at the same time, so its moving the TV shows from I to J at the same time.  The only thing is, its not removing the shows from I, its leaving a copy there.  Maybe it will remove them on completion (Ive started a large copy), though I dont know why it didn't immediately go to J since that was the file placement rule.

[BuddyRich]

Actually once the amount of data in TV surpassed Movies it started to put TV on H as well.   The balancer is running (and it says its writing TV from H ...) but should it not have went to the drive in the rule in the first place?  This is a lot of extra writing happening on the disks.

[BuddyRich]

So I turned off Balanacing completely and unchecked all balancers on the balancing tab.  it started respecting the file placement filters...   Ive gone ahead and bought DrivePool.  It looks like it will accomplish what I want.

 

 

What will happen once one of the drives is full?  Will it write to the drive with the most free space by default?

 

 

[Christopher (Drashna)]

If you're using File Placement Rules with the Ordered File Placement balancer, then you absolutely need to do the following:

• UNCHECK "File placement rules respect real-time file placement limits set by the balancing plug-ins."
• CHECK "Balancing plug-ins respect file placement rules."
• UNCHECK "Unless the drive is being emptied."

 

This will allow both features to work "in harmony". As mentioned in the notes for the balancer:

http://dl.covecube.com/DrivePoolBalancingPlugins/OrderedFilePlacement/Notes.txt

So I turned off Balanacing completely and unchecked all balancers on the balancing tab.  it started respecting the file placement filters...   Ive gone ahead and bought DrivePool.  It looks like it will accomplish what I want.

 

 

What will happen once one of the drives is full?  Will it write to the drive with the most free space by default?

If the balancing settings are enabled, once one of the drives gets to the limit (90%, IIRC), then the Ordered File Placement balancer should change the real time limits and force files to be placed on the next disk in the list. 

 

 

Otherwise, the default strategy is to place files on the disk with the most available free space (absolute, not percentage. 

[BuddyRich]

So here are my settings now:

 

(On the balancers table I only have the Ordered File Placement enable and my filters was a the first post).

 

Its working, but its still not doing automatic balancing (that option is turned off).  The icons show that my pool needs balancing, but why is the 3rd drive showing it?  It currently only has TV Shows, so if the balancer respects the filter it shouldn't need to be balanced.  What will happen if I tell it to go ahead and balance?

 

One other problem Ive noticed is that it hit the 90% full on the first drive, and spilled over to the next drive, but its separated contents of folders rather than keeping them together.  I have a movie folders with the large MKV in it and smaller .nfo and thumbnail jpgs used for Kodi.  It put the MKV and one drive, and the rest of the metadata under the same folder on another drive.  Im sure Kodi will see it all as one but is there anyway to stop this?

[Christopher (Drashna)]

Well, I'm glad to hear that it does seem to be working better.

 

As for the balancing, the file placement rules are handled by the balancing code, so you'd either need to run a pass manually, or you need to set it to automatically occur. 

 

As for why the 3rd disk is showing an issue with the file placement rules, is that it may contain files that that need to be moved off of the disk in question to fulfil the rules.

 

As for the folder, unfortunately, this may happen. The balancer does a best effort to keep the files together, but it may not always.

If you absolutely must force this, then you *can* use the file Placement rules to enforce this. Aside from that, change the limits for the Ordered File Placement balancer and hope it doesn't happen again. 

[BuddyRich]

Well I spoke to soon.  I was copying files back to the pool and as soon as Drive I matched Drive J in usage it then just round-robined between them for the rest of copy, rather than using the ordered placement balancer.  Im guessing I wanted the automatic balancing to be on at that point so it would actually use ordered placement?  Otherwise, even though it was the only balancer enabled, the default is to write to the drive with the most free space regardless of setting since I was actually using a balancer?

 

When the copy was done, I manually ran the balancer to try and clean up the extra movies files on J.  First time I tried to balance it said it couldn't move ~4TB of data so it didn't run.  I don't know why it said that, it really only had to move about 300GB from Drive J to I since about 2TB was actual TV shows that should have just stayed on J according to file placement (and everything on I was fine and didn't need to moved either).  It was like it was trying to move all stuff oiff I and J and balance it again, shouldn't it have just done a "difference" rather than trying to balance it all from scratch?

 

As a workaround I set movies to just be on H and I only and ran it again.  Its not showing anything in performance, but balancing says it is moving from drive J: to I: and I can see a bunch of folders moving from J to I, though its leaving empty folders on J so it seems to be working.

 

Is there any harm if I manually remove the empty folders that are left under J: after the balancing?

[Christopher (Drashna)]

Yes, please try again with the balancing enabled. It should (hopefully) help prevent the "round robining".

 

If it does persist, enable logging and reproduce:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

 

And if you're moving empty folders, it should be fine.

[BuddyRich]

So I had a bit of time to play with this.  I did a rebalance and it seemed to move more smaller files creating duplicate folders across pool drives rather than keeping folders of files together, which is funny because I have space left to accomodate at least some of the bigger files to keep some folders together, the balancer just didn't do it.

 

I updated to the 2.2 beta version as well, but Im still having the issue.  Also I am getting this when it balances (I had this with 2.1 release as well).  Why is it saying I have 4TB of data to balance?  Thats pretty much all of the files on the drives.  Some are in the right place because of file placement rules restricting them to certain drives.

 

 

 

 

 

[BuddyRich]

So I added 2 more freshly formatted empty 2TB drives to my pool.  F: and G:.

 

Set a file placement filter on the Music folder to go to drive F: and only F:, did a large copy to Music on the pool and it still balanced between F: and G: (the other 2GB drive) as they were both equally empty so 150Gb of each was on each drive.  This despite both the Ordered File Placement Balancer being the only balancer enabled and the File Placement Filter that was set to go to only drive F:.

 

That said the balancer was running during the copy, and after it completed, it eventually moved everything that was placed on G: off of G: to F: as per the filter but it didn't do either the filter or ordered placement balancer realtime and just the the round robin thing.  Is it supposed to be realtime or is it only after the fact?

[BuddyRich]

So Ive reproduced the problem.

 

 

On the two new drives F and G, I copied a folder of .mp3s to my pool (L:) under folder Music.  Music has a filter that says music goes to F: and only F but it still copies it all over the drives in the pool, despite the ordered file placement balancer AND the file placement filter, its like its just ignoring it and doing round robin. 

 

Actually thats not entirely true, the balancer is running at the same time, and it eventually gets to the right place, but should it not respect the filter first and only copy to drive F:, rather than all over THEN balance?

 

At any rate I had logging enabled when I did this, where do I submit the log as I think its a bug?

[BuddyRich]

So I submitted the logs via this procedure:

 

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

There was no spot to put my email, how can I followup?

[Christopher (Drashna)]

Thank you for grabbing the logs. 

 

I've flagged the issue for Alex (the Developer), and linked the files. You can follow the issue here:

https://stablebit.com/Admin/IssueAnalysis/24909

 

However, I'll definitely post here, to make sure. 

[BuddyRich]

Is there any update on this?  If the balancing isn't immediate this a lot of wear and tear balancing after the fact.  I need OFP because I am using SnapRaid for parity.

[Christopher (Drashna)]

Unfortunately, no, there isn't. 

 

Alex has been focusing on StableBit CloudDrive, in a large part because we "just want to get it out the doors".  Once that's done, we do plan on hitting a lot of "back issues" like this and getting them resolved.