Jump to content
  • 0

Ordered File Placement Not Working


rmahin

Question

Hi All, 

 

I am still evaluating this product and am very new to it so forgive me if I'm overlooking something simple. I, like many user out there, am attempting to use Drivepool and Snapraid, so I am not using any duplication. I have read the best approach to using both is to disable all balancing, and use the Ordered File Placement plugin to fill up one drive at a time. However, I also like to keep some of my stuff on specific drives and have setup file placement rules for those directories. Namely, I want TV Shows to be placed in Drive1 and Movies in Drive2 using the file placement rules, and everything else go to Drive3(and 4, 5, etc etc) as per the Ordered File Placement drive priority list.

 

I thought I had this setup properly, the overview page seems to indicate what I would expect. 

 

post-2268-0-95621800-1450284216_thumb.png

 

My understanding of the orange arrows on the bottom was that they indicate where new files will be placed. However, 90% of the time, any new files added to the pool will be added to G:\, assuming it's because it has the most available space. Even Movies, which should go to Drive2 will go to Drive1. I cannot seem to pinpoint exactly when files are placed correctly, and while it does seem to happen occasionally, it is very rare.

 

Here are screens of what I hope are all the relevant settings:

 

post-2268-0-54148900-1450284482_thumb.png

 

post-2268-0-86104600-1450284664_thumb.png

 

post-2268-0-67317700-1450284668_thumb.png

 

post-2268-0-82909500-1450284670_thumb.png

 

post-2268-0-47099400-1450284673_thumb.png

 

post-2268-0-92968600-1450284675_thumb.png

 

Hopefully I've just overlooked something simple and you can point me at an easy fix, and let me know if there's any more information I can provide. 

 

Thank you in advance for any help you can offer!

Link to comment
Share on other sites

25 answers to this question

Recommended Posts

  • 0

Sorry for the delay in getting you!  I thought I had replied to you already, but it looks like I didn't. I apologize for that!

 

I think the issue here is that the balancing is set to "do not balance".  Specifically, the File Placement Rules are part of the balancing system, and require a balancing pass to get things moved around. And from your initial post, it looks like balancing is disabled.

 

 

If you're using FlexRAID/SnapRAID/etc in conjunction with the pool, it maybe problematic for existing files... as you may need to enable this at least temporarily to get everything moved and placed properly. 

 

 

 

As for the trial period, I've created a trial extension for you, and have sent you an email for it. It will contain an Activation ID that you can use to extend the trial until Feb 15th. 

Link to comment
Share on other sites

  • 0

Thank you both for your replies. I will trust your guys' opinion on the quality of support here. I have come up with a (rather obvious really) workaround that should fulfill most of my needs, and that's to simply to add file placement rules for all top-level directories. Only parts that could e a  bit annoying will be:

  • adding new drives: I will have to edit all rules for all top-level directories that can use the new drive, instead of just just Movies or TV Shows directory. 
  • adding new top-level directories: I will have to create new rules for the added directory

I still think something odd is going on with the Ordered File Placement plugin -- at the very least the current behavior seems quite counter intuitive, but I think adding the rules will achieve the desired result. And while this solution is not quite as automated as I would like, both scenarios can be can be limited quite easily so the maintenance time would not really be a huge issue

 

So I do have a suitable workaround it seems, but would still like to know why it's not behaving as I would expect. 

 

Thanks in advance.

Link to comment
Share on other sites

  • 0

Is your Pool organization 100%? If not, then you can instruct it to re-balance. Perhaps you experience this behaviour because you do not have automatic balancing and, though I am by no means certain, I think you need balancing for the Balancer Plug-Ins to actually work. I have Balance immediately and not more often than every is unchecked.

 

Edit: Never mind, that won't be it. Christopher will come and help you out but, well, he may be enjoying some time off given the season.

Link to comment
Share on other sites

  • 0

No worries on delay for reply -- other people attesting to the quality of service here, and your response is enough to give me faith in the product :)

 

Enabling balancing definitely did the trick! For some reason did not make the connection that these operations were considered "balancing", so thanks for that hint! 

 

So, I've spent some time with this again, I have updated my balancer settings to the following and would like to know if the behavior I'm seeing is expected. 

post-2268-0-77038100-1451292381_thumb.png

 

Using the same example as in OP, I currently have ~1.3TB available on Drive1, ~1TB available on Drive2, and .9TB available on Drive3.  

Again, Drive1 is designated for /TV Shows, Drive2 for /Movies, and Drive3 is set as priority drive for Ordered File Placement.

Assume I am talking about everything relative to the poolpart directories on each drive.

 

Scenario 1:

Add a new file to root storage pool directory

Result:

The file is copied to the root directory on Drive3 as expected, YAY!

 

Scenario 2:

Add a new file into /TV Shows in storage pool

Result:

The file is created in /TV Shows on Drive1 as expected, no superfluous items are created, YAY! 

 

Scenario 3:

Add a new file into /Movies in storage pool.

Result:

Creates a /Movies directory on Drive1 with the new file. When balancing runs next, it moves the file to /Movies on Drive2, leaving /Movies on Drive1 empty.

 

Question 1:

How long does it take for balancing to start? Even though I have "balance immediately" checked, it does not always happen right away.

Question 2:

Why does it first place the file on Drive1 at all? My only hunch is because Drive1 has the most available space. 

Question 3:

Is there a way to have the file simply go directly to Drive2? It seems like it should not have to wait for balancing to kick off. 

Question 4:

If there isn't, is there a way to have it remove the empty directory after balancing occurs? I realize its not really causing any harm, but it seems like there is potential for a rather large number empty directories scattered among the drives that I would really like to avoid if possible. 

 

Thank you in advance for the help!

Link to comment
Share on other sites

  • 0

Well, thanks!

 

 

And that is definitely understandable. The balancing system has significantly grown and improved with time, and does a lot more that it was initially designed to. 

 

 

 

As for your settings, those do look good.

The file placement settings (at the bottom of the general page) are set to what they need to be to ensure they work with the file placement rules (check the notes for the balancer for details):

http://dl.covecube.com/DrivePoolBalancingPlugins/OrderedFilePlacement/Notes.txt

 

 

However, I would recommend setting the ratio slider higher, and maybe enable the checkbox under it. This may help make the balancing a bit more aggressive.  That may be a good thing for your setup. 

 

This basically deals with the first question. The Ratio determines when the balancing is triggered. As files get "out of place", it lowers the ratio. Once it hits the threshold, it will trigger a balancing pass. The higher the threshold is set, the more aggressively the system will balance.

Additionally, enabling the "or this amount to be moved" option is a good idea if you have a very large pool, as a 100GBs may be too small to trigger the balancing aggressively, as your pool grows larger.

 

 

 

As for Question 2:

Yes, that's precisely why. The default placement strategy is to place files on the disk with the most (absolutely) free space. If there are multiple or no suitable drives, this is what the software defaults to. 

 

 

Question 3:

This is controlled by the "Real time placement limiters", as set up by the Ordered File Placement (or other) balancers. 

In theory, the file placement rules should trump this, if you have the "file placement rules respect ...." option disabled (which you do).  If it's not sending the files directly to that drive, that may be an issue.  

 

 

Question 4:

Not really. This is normal. Unfortunately, there isn't a good way to really handle this. 

Partially, because we store the duplication status of the pool in the folder structure (as super hidden file system objects, called alternate data streams), so we need at least part of the folder structure to be on multiple disks if duplication is enabled.

However, it really shouldn't cause any sort of harm. 

Link to comment
Share on other sites

  • 0

Ok awesome! Thanks very much for the explanations and the tips, I really appreciate it. 

 

 

In theory, the file placement rules should trump this, if you have the "file placement rules respect ...." option disabled (which you do).  If it's not sending the files directly to that drive, that may be an issue.

 

The file placement rules do not seem to be trumping this if that's something you guys wanted to look into, but I agree it is not very high priority or concerning as the balancer will sort it out. 

 

Really, thanks again :) 

Link to comment
Share on other sites

  • 0

Sure thing, here is the service log. 

 

Steps taken:

  1. Started drivepool service
  2. Started file system logging
  3. Made a new directory inside the /Movies directory in the drive pool (this step is not necessary to reproduce the problem -- get the same results creating a single file and I only created a new directory to have both a new directory and a new file in the log)
  4. Made a new file inside the newly created directory
  5. Manually started a re-balance
  6. Ended file system logging

Result:

The newly created files get placed on the drive designated for TV Shows, and get moved when balancing is triggered.

 

If I did something incorrectly, or need to provide any more information just let me know, thanks again for looking into this :)

DrivePool.Service-2016-01-02.log

Link to comment
Share on other sites

  • 0

Okay, I've flagged this for Alex, so he can take a look at it.

https://stablebit.com/Admin/IssueAnalysis/22890

 

However, it may take a while to get to, sorry. 

 

 

ALso, what rules are you using specifically? 

And what OS are you using, and what version of StableBit DrivePool?

Link to comment
Share on other sites

  • 0

No worries on the delay, you've got me running with two different possibilities so thanks again! 

 

I'm using 2 file placement rules

\TV Shows\* -> G:\

\Movies\* -> H:\

Allow files to be placed on another drive when its 98% full. 

 

Ordered File Placement set with I:\ as highest priority. 

 

Pictures for all settings are in OP.

 

Operating System:

Windows 10 64 bit

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.10586 N/A Build 10586
 
Drivepool Version
2.1.561
 
As always, let me know if ya need anything else. Thanks again for taking a look!
Link to comment
Share on other sites

  • 0

Thanks for posting the info.

 

Just in case, could you install the latest beta build, and see if that helps? There are some changes to balancing code, and it may help.

http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.659_x64_BETA.exe

 

Additionally, there are some auditing tools in there, that may help out with identifying the problem section (or at least, list where the files actually reside).

Link to comment
Share on other sites

  • 0

Well, you can use the "dpcmd" tool to see where the files are actually ending up:

http://community.covecube.com/index.php?/topic/1587-check-pool-fileparts/

 

Just juse "x:\path\to\folder" instead of just the pool letter, to see just that folder.

 

And if it's still round robining the files, could you enable file system logging and let this occur:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Link to comment
Share on other sites

  • 0

Did as requested, made a new directory in P:/Movies

 

C:\>dpcmd check-pool-fileparts "P:\Movies\New folder"
 
 
dpcmd - StableBit DrivePool command line interface
 
Version 2.2.0.659
 
Detail level: Inconsistent File Duplication
 
Listing types:
 
  + Directory
  - File
  -> File part
  * Inconsistent duplication
  ! Error
 
Listing format:
 
  [{0}/{1} IM] {2}
    {0} - The number of file parts that were found for this file / directory.
    {1} - The expected duplication count for this file / directory.
    I   - This directory is inheriting its duplication count from its parent.
    M   - At least one sub-directory may have a different duplication count.
    {2} - The name and size of this file / directory.
 
Scanning...
 
+ [2x/1x I] P:\Movies\New folder\
 
Summary:
  Directories: 1
  Files: 1 0 B (0 B)
  File parts: 1 0 B (0 B)
 
  * Inconsistent directories: 0
  * Inconsistent files: 0
  * Missing file parts: 0 0 B (0 B)
 
  ! Error reading directories: 0
  ! Error reading files: 0
 
Also attached the log with URL for this topic in the description. If there's a better way to link it, let me know
Link to comment
Share on other sites

  • 0

Well, could you use the "4" level, for the folder in question, specifically. This should print out the exact location for all of the files. 

 

Specifically try using this command:

dpcmd check-pool-fileparts "P:\Movies\New folder" 4 > Pool.log

This will export the list to a file, and then please upload that file. 

Link to comment
Share on other sites

  • 0

sorry for the long delay here. 

 

Alex has identified the specific issue that caused this, and it has been fixed in the latest beta. 

 

Either wait a bit longer for it to "go live", or you can find the patched version here:

http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.762_x64_BETA.exe

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...