Jump to content
Covecube Inc.
  • 0
Beach

File placement based on folder

Question

Hi, im trying to figure out if its possible to make a rule so that folders and all files in the subfolders are placed on the same Hard drive together instead of each file in a folder placed on random hard drives. Ie

 

 

C:\

     Folder 1

     Folder 2

         Folder A

         Folder B

     Folder 3

         Folder C

 

 

In the example above all the files in Folder A will be placed on the same HD(doesn't matter which but all together). Folder B files all on the same HD. etc etc

 

 

Reason for this is if I have a HD failure instead of missing random files from random folders its a lot easier for me to say "ok Folder A is gone and needs to be restored"

 

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

I was just wondering this myself and had already looked at the ordered file placement. My question to this is similar but slightly different than the way that balancer states it works.

 

For me let's assume I'm transferring media and specifically TV shows. So I'm gonna transfer 6 seasons worth of "TV Show A" and 10 seasons of "TV Show B". If I read the balancer description correctly it's going to completely fill one drive then move onto the next one which means by the time it gets all the way down to "TV Show K" it may end up splitting that directory simply because it's out of physical space.

 

So now my question becomes is there any way to keep and entire folder/sub folder together without filling each disk to full then moving on? Like the OP, I don't care which disk it uses as long as each series is all together. And to be completely honest, I like the way the default does it where let's assume you have 100 movie files and 5 disks, it will put one movie on each of the five drives then start over to keep everything balanced. That works great for individual files as it really doesn't matter where they are.

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post
Share on other sites
  • 0

I'm sure it's more of a pain than what's it worth. I'll leave mine how it's set. Just thought it was worth asking.

 

From our side convo, about 27TB transferred so far and still a bunch to go!

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post
Share on other sites
  • 0

unRaid has a concept of 'split level' where you can specify upto what level files/folders should be placed on the same drive. I don't see anything similar in DrivePool but maybe I'm not seeing that option. 

 

e.g. if I have

 

movies\

     movie1

     movie2

 

tv\

   show1\

      season1

      season2

   show2\

      season1

      season2

 
is there a way to setup things so that files in a season or movie folder are never split across drives? Or if a season has to split then the folder is recreated?

Share this post


Link to post
Share on other sites
  • 0

 

unRaid has a concept of 'split level' where you can specify upto what level files/folders should be placed on the same drive. I don't see anything similar in DrivePool but maybe I'm not seeing that option. 

 

[...]

 
is there a way to setup things so that files in a season or movie folder are never split across drives? Or if a season has to split then the folder is recreated?

 

 

To implement something like this would require a complete re-write of the balancing engine, which is by no means a small project.  It is possible that we could do something like this in the future and make it much more robust, but that's something we plan on doing right now.  

 

As for enforcing placement, you can use the "file Placement rules" to force this.  But you'd have to micromanage the pool to do this properly.  

Share this post


Link to post
Share on other sites
  • 0

To implement something like this would require a complete re-write of the balancing engine, which is by no means a small project.  It is possible that we could do something like this in the future and make it much more robust, but that's something we plan on doing right now.  

 

As for enforcing placement, you can use the "file Placement rules" to force this.  But you'd have to micromanage the pool to do this properly.  

 

This is the split level I'm referring to - https://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Split_level

 

I'm not sure how the current balancer is implemented, but this doesn't seem that hard computationally. Since it just needs to look at the path depth and decide if a new disk should be used. The current file placement algorithm must do this as well when it decides how to place files, it just looks at different criteria. I have taken a look at writing my own balancing plugin but there doesn't seem to be enough info passed to the plugins and in any case these are called after the files have been placed initially. Not when they are being copied to the pool and DrivePool decides where they go, correct?

 

I'm afraid with the defaults, I'll end up with files in the same folder split across multiple disks (and this can also need all of them to be spun up). Other people also seem to have complained of this.

 

e.g - http://community.covecube.com/index.php?/topic/2153-crashplan-restore-is-a-nightmare-if-using-default-drivepool-placement/

 

IMO this should be a bit higher priority. If this can be done via a balancer plugin I'm willing to help out.

Share this post


Link to post
Share on other sites
  • 0

yes, I did understand what you meant about the folder management.  And yes, it is a fairly common request. 

 

 

As I said, a change like this to the balancing engine would require a complete rework of the code. And part of the problem here is that it wouldn't just be the service, but in the kernel driver as well.  The more complex the code because, the more of an issue it because for the kernel code.   Since this needs to be fast and quick,  adding significant complexity to the code could (and more than likely would) add a significant overhead (eg slowdown) to the pool. 

 

And yes, it is something it is coming up more and more frequently, and likely something that we'll need to deal with (sooner rather than later, most likely).  But it's going to be a significant investment of time to do so. 

 

 

 

That said, this is something that I do plan on pushing, because of a) how frequently it does get brought up (though it's like once every few months though), and that it would add a lot of functionality to the software.

Share this post


Link to post
Share on other sites
  • 0

Thanks, so as I understand it this cannot be done by just a file balancer plugin and needs core changes, so there's no point in trying to write my balancer for this. I hope Alex gets the time for this but there are probably more important things in the pipeline. btw is there a roadmap for your products?

Share this post


Link to post
Share on other sites
  • 0

Given how complex the balancing system is getting, a complete overhaul isn't out of the question (but not up to me). 

 

As for a road map, not really.  We do have some info here: 

http://wiki.covecube.com/Development_Status

 

 

But right now, we're finishing up StableBit CloudDrive (some major issues for it), and then we plan on hitting the backlog of bugs for StableBit DrivePool and then StableBit Scanner.  After that, we plan on adding more features, and/or change things. 

 

And no, we don't have a timeline for this.

Share this post


Link to post
Share on other sites
  • 0

Given how complex the balancing system is getting, a complete overhaul isn't out of the question (but not up to me). 

But right now, we're finishing up StableBit CloudDrive (some major issues for it), and then we plan on hitting the backlog of bugs for StableBit DrivePool and then StableBit Scanner.  After that, we plan on adding more features, and/or change things.

 

​​Can you provide an update? It looks like CloudDrive has been "finished up". Are new features for DrivePool developed now?

 

I'm not using the product yet. The current state with the Ordered File Placement Plugin compromise is probably not enough for me to spend money on it. I plan on using DrivePool for a media library without duplication. Having the same TV show with all seasons on the same harddrive is a must-have for me.

Ideally it should look like unRAID Split Level 2 (e.g. TV shows can be on different drives but one specific tv show is only on one drive). That way when one hard drive dies I know exactly what went missing so just need to rip the DVD/Blu-rays of the affected shows again.

Share this post


Link to post
Share on other sites
  • 0

Alex has been going through the backlog of bugs.  And is almost done. 

 

We hope to have a new release out soon.  As for new features, not quite yet.  I think that we want to get a stable version out, and then start working on any more new features.

Share this post


Link to post
Share on other sites
  • 0

​​

We hope to have a new release out soon.  As for new features, not quite yet.  I think that we want to get a stable version out, and then start working on any more new features.

 

Thanks for the reply.

 

Unfortunately Alex just responded today and he personally doesn't see any need for this feature. As the sole developer if he doesn't stand behind the feature, it's not high on the priority list. This is very disappointing.

 

Here is the quote:

"​Grouping files "together" has been requested a bunch, but I personally, don't see a need for it. There will always be a need to split across folders even with a best-effort "grouping" algorithm. The only foolproof way to protect yourself from file loss is file duplication."

​

​I don't understand what's so difficult to get here. DrivePool makes a big volume out of many hard drives. I'd wager a majority use this software (in part) for an easy to manage media library. Why else would you have multiple Terabytes of data? For text documents? How is it not logical then to get folder split levels to make replacement easy after a hard drive fails?

​

It also doesn't make any sense to me to enable duplication for media that I physically own and can get back anytime by ripping the DVD/Blu-rays again. It is simply wasted hard drive space.

Share this post


Link to post
Share on other sites
  • 0

a counter view  :P

 

I find it hard to understand using DP without duplication - kind of the point as you are looking for redundancy - having one big drive with minimal or selective duplication is the way to problems as you describe if you dont know what drive has what - you "almost" might as well have a JBOD or Raid0 

 

I dont care which of my files are on which drive as a failure is not a big issue - unlike potentially with raid - one of the main reasons i also use DP - minimal rebuild time much less chance of a second disk failing when re-dupping my files - if i want more than one level of redundancy i can go to x3,or x4 duplication etc or as i have done two pools. (I have used raid for years and still have a QNAP with 8*4TB that i still use daily as it has a few advantages so well aware of the pitfalls and benefits and still i think on balance DP is a better solution)

 

If you must micro manage your files - use the balancers and specify what goes where - if its media files they are not going to move around once the pool is balanced after they are added - unless you keep tinkering with the settings! 

 

As an alternative copy the files directly into the pool part directory on a specific drive - much more likely to stay together then and you know where they are.

 

Oh and its easy to say you can re rip your blu-ray collection if things go south until you have to do it and spend days doing it even with a relatively small collection.

 

Have fun

 

puts on tin hat  :D

Share this post


Link to post
Share on other sites
  • 0

I find it hard to understand using DP without duplication - kind of the point as you are looking for redundancy

 

I'm not looking for redundancy, simply for a way to group drives together to one big volume.

What I like about DrivePool is that even if the software is discountinued or a hard drive fails, I still have a normal folder/file structure on each drive. I don't want any RAID because of the rebuild issues and because it's not a backup.

 

 

Oh and its easy to say you can re rip your blu-ray collection if things go south until you have to do it and spend days doing it even with a relatively small collection.

I agree that it would take a lot of time but I also have a full backup on Amazon Cloud Drive Unlimited. With my 500 Mbit connection I can restore any lost media almost as fast as I can fill a new hard drive.

AmazonCloudDrive.png

 

 

 

As an alternative copy the files directly into the pool part directory on a specific drive - much more likely to stay together then and you know where they are.

I thought about that but I don't like such micromanagement. Simply dropping data into the pool is preferable for me.

 

For now I can live with the Ordered File Placement plugin. I've disabled all other balancers.

Share this post


Link to post
Share on other sites
  • 0

 

​I don't understand what's so difficult to get here. 

 

 

Oh man...  Just becacuse it's a simple idea doesn't make it simple to implement.  In fact, it's a running joke "internally", because there are a few times that Alex has said something will be simple, and six months later .......  Reparse point support, for instance. Or, file placement rules.  And StableBit CloudDrive, as well.  And so on. 

 

That said, we do talk about this a LOT internally, because it does come up a lot.   While I agree with Alex, that's not my job.  People want this, so I will keep on bringing it up, even if I never actually use it.  

 

 

But the thing is, the way that StableBit DrivePool is designed, the goals for it, from the ground up, are kind of counter to grouping the specific files, and micromanaging.  

 

Additionally, because we are a very small company, it means that we have limited resources.  Which means things HAVE to be prioritized.  And things that we (as a company) don't want will naturally get pushed to the back burner (or canned).   But then again, that's why I keep on bringing up this stuff.   If I keep bugging Alex about it, it will be on his mind, and he may come up with a good way to implement it.  

Share this post


Link to post
Share on other sites
  • 0

Oh man...  Just becacuse it's a simple idea doesn't make it simple to implement.

Seems like you misunderstood me and took my quote out of context.

 

The "I don't understand what's so difficult to get here" is a direct response to Alex's post "[...] but I personally, don't see a need for it. There will always be a need to split across folders even with a best-effort "grouping" algorithm".

 

If you talk about this feature as much as you say, this is either a poor choice of words on his part or he doesn't get the Folder split level feature at all. Because in my opinion no, there is not always a need to split across folders. My use case with only the Ordered File Placement Plugin activated proofs it. And as I said he's the only developer, if he doesn't stand behind the feature or can't even grasp why it is needed, it's not high on the priority list.

 

What I'm not saying is "this is so easy to implement, why hasn't it been done yet in all those 5 years?".

Share this post


Link to post
Share on other sites
  • 0

Alex's post: but I personally, don't see a need for it. There will always be a need to split across folders even with a best-effort "grouping" algorithm".

I have to agree with Wiidesire on this one. To be blunt, Alex is so completely wrong on this I don't even know where to begin. I currently use unRAID right now even though I own two DP licenses. My reason? Folder split level. I don't have the need or desire to go into hidden folders every time I want to drop new episodes of TV Show X into my pool. In this day and age, it's silly to expect the user to do that. I would prefer to be running Windows and DP since the rest of my network is all windows but I will not be transferring all my data back till this folder split level is resolved. That's not a threat in any way shape or form. I even bought a cloud drive license even though I don't use it. I bought it to support Alex and the continued development of SB products.

 

I usually stay pretty quiet on here but Alex saying he doesn't see a need for it kinda pisses me off. Makes me curious if he forgot who pays for this software. That would be us, the customers. Again, to be blunt, I honestly don't care if Alex sees a need for it or not. If it gets requested frequently, like Chris has said it does, then Alex needs to address it. It's that simple.

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post
Share on other sites
  • 0

 While I don't need this myself (I don't see the need for it, personally), I do understand the reasoning behind the desire.  

 

Which is why I keep pushin on this topic.

 

 

As for the quote, there may be some hostility/bluntness there, due to the context.  

Share this post


Link to post
Share on other sites
  • 0

Well, I'm glad to hear it, though I suspect that you may like the next product or two that we have planned. 

 

 

And I do still want to get the "folder" thing in there.  As I've said, we've talked about it a lot, internally. 

Share this post


Link to post
Share on other sites
  • 0

Uhm, I actually want DP to be as simple and fast as possible and would prefer not to have fancy placement stuff in there. But then, I consider the File Placement Rules evil as well. My fear would be that as DP does more, it becomes more vulnerable to bugs and user mishaps. I wonder (but I may be well off here) whether using seperate HDDs and then defining libraries in Windows would not be a more logical solution for the suggested function here.

 

One of my main concerns is that (and I have looked at the UnRAID link provided above briefly) either DP can not guarantee placement as expected, causing issues when one expects to recover easily, or must check and produce error reports. Let's say for instance that you want a certain folder on one HDD, the folder size is 3TB and the biggest HDD is 2TB. What should it do? It may then spill over to others (this is what happens with File Placement Rules if I am not mistaken) but unless this is reported clearly, a user may still be off as bad as with random placement (where the user knows placement is "random"). If it is reported clearly then the user must still micromanage. DP may also say there is not enough space but you might have a 10TB Pool with 5TB free and still run into this. Another concern is that DP would have to become more context-aware when placing files or balancing and, when free space becomes limited, may need to reshuffle entire folders to make room for an addition to another. The UnRaid document alludes to such issues as well with their implementation of split levels (and as far as I can tell it does not do a reshuffling at all but errors out on free space even if in the aggregate there is a lot available).

 

I prefer DP to offer a single virtual drive and manage placement as it seems fit (use least occupied HDD or fill all equal %-age), run as unattended as possible and, if you have duplication, it will recover itself (for instance if you have a hierarchic Pool with one base-Pool/string as Amazon Cloud Drive Unlimited with a 500 Mbit connection and local HDDs as the other base-Pool/string). If it becomes bloated with user options and suffers in stability then that would, IMHO, be a bad thing.

 

Anyway, just my 0.02$ worth.

Share this post


Link to post
Share on other sites
  • 0

Well, I'm glad to hear it, though I suspect that you may like the next product or two that we have planned.

If it's Stablebit Telemetry Remover, I wanna pre order!!

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post
Share on other sites
  • 0

If it's Stablebit Telemetry Remover, I wanna pre order!!

 

 

Sent from my iPhone using Tapatalk

 

haha!  

 

Well, we don't generally collect telemetry ourselves, outside of the activation information. 

 

And there are plenty of tools to do this for Windows 10. 

 

But you can see the planned stuff here:

http://wiki.covecube.com/Development_Status#Future_Products

 

 

 

 

Uhm, I actually want DP to be as simple and fast as possible and would prefer not to have fancy placement stuff in there. But then, I consider the File Placement Rules evil as well. My fear would be that as DP does more, it becomes more vulnerable to bugs and user mishaps. I wonder (but I may be well off here) whether using seperate HDDs and then defining libraries in Windows would not be a more logical solution for the suggested function here.

 

One of my main concerns is that (and I have looked at the UnRAID link provided above briefly) either DP can not guarantee placement as expected, causing issues when one expects to recover easily, or must check and produce error reports. Let's say for instance that you want a certain folder on one HDD, the folder size is 3TB and the biggest HDD is 2TB. What should it do? It may then spill over to others (this is what happens with File Placement Rules if I am not mistaken) but unless this is reported clearly, a user may still be off as bad as with random placement (where the user knows placement is "random"). If it is reported clearly then the user must still micromanage. DP may also say there is not enough space but you might have a 10TB Pool with 5TB free and still run into this. Another concern is that DP would have to become more context-aware when placing files or balancing and, when free space becomes limited, may need to reshuffle entire folders to make room for an addition to another. The UnRaid document alludes to such issues as well with their implementation of split levels (and as far as I can tell it does not do a reshuffling at all but errors out on free space even if in the aggregate there is a lot available).

 

I prefer DP to offer a single virtual drive and manage placement as it seems fit (use least occupied HDD or fill all equal %-age), run as unattended as possible and, if you have duplication, it will recover itself (for instance if you have a hierarchic Pool with one base-Pool/string as Amazon Cloud Drive Unlimited with a 500 Mbit connection and local HDDs as the other base-Pool/string). If it becomes bloated with user options and suffers in stability then that would, IMHO, be a bad thing.

 

Anyway, just my 0.02$ worth.

 

 

we want all of our products to be simple, and very "hands off".   But we do want options too.  It's a hard balance to find, and one that many companies either never do, or don't bother to.

 

 

And you're right on the behavior.   And part of that is that it was never designed to work this way, and file placement took a lot to get working, even "as is".   

 

And Alex is hesitant about adding this sort of stuff, because it's counter to the original designs of StableBit DrivePool, and "outside of his use case".  

 

Neither is a good reason to not add it... but given the coding complexity to add this, it's not insignificant, and since Alex is the one developing it .... it's ultimately his call.   

 

But, again, that's why I keep in trying to push this.  I'm the "customer service guy", which means that it's literally part of my job to push stuff like this, because it *is* highly requested.  

 

 

(also, sticky note to push this again, hard at the next video conference we have) 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...