Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

StableBit DrivePool - Controlling Folder Placement


  • Please log in to reply
53 replies to this topic

#1 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 18 September 2013 - 06:08 PM

I like writing these posts because they give me feedback as to what the community is really interested in. I can see that my last post about the Scanner was not very interesting, it was probably too technical and there's probably not much to add to what I've already said.

 

Well, this time let's talk about StableBit DrivePool. In particular, I'd like to talk about DrivePool beyond 2.0.

 

Controlling Folder Placement

 

I think that I have a few great ideas for DrivePool 2.1+ but some of them depend on the ability to control folder (or file) placement, per pool part. I've kind of hinted at this capability in the thread that talked about taking out per-folder duplication, but I think that I've figured out how we can make this work.

 

What I would like to be able to do in future versions is to give you guys the ability to associate folders with one or more disks that are part of the pool. So that any files in those folders would be stored on those pool parts only (unless they're full).

 

This should be trivial to implement on the file system level, but the balancing framework would need to be enhanced to support this, and I think that I've figured out how to make that work.

 

Theoretically, you should even be able to use wildcard patterns such as /Virtual Machines/Windows* to associate all of those files with a group of pooled disks.

 

What do you guys think, is this worthwhile doing?


  • meeldilla and Tardas-Zib like this

#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 18 September 2013 - 07:06 PM

Because it's kind of wordy.... you mean to be able to control what files go to which disk in the pool, basically, correct?
If so, I know that I would definitely be interested, I'm sure there are others that definitely would be as well.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 19 September 2013 - 01:38 AM

Because it's kind of wordy.... you mean to be able to control what files go to which disk in the pool, basically, correct?
If so, I know that I would definitely be interested, I'm sure there are others that definitely would be as well.

 

I can be a bit wordy by my very nature. But yes, that's exactly what I'm talking about, controlling which files go onto which pool part.

 

And, in the future, a pool part may not necessarily represent a single local physical disk, which would make this even more interesting :)


  • Christopher (Drashna) likes this

#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 19 September 2013 - 10:22 PM

Okay, and yes, I know I'd be very interested in this. And I'm sure I'm not alone.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 19 September 2013 - 11:32 PM

This is something that I have hoped for, for a very long time. I really have never been keen on having things scattered around the pool. Especially with music... 10 album tracks scattered over 5 or 6 drives.. But then I'm probably a bit O.C.D.

 

I'm sure for people who have various devices streaming to different rooms this must be a good thing. Knowing that all the Disney films are all on one hard drive for the rug rats, and the teenage daughter can watch her twilight knowing its on a separate drive so no risk of intensive I/O and so on. And yes i know that this could be achieved by organising multiple pools. However when you get to 13 drives and around 22TB of data creating new pools seems like a hassle. 

 

First thought is that this would eliminate the need.

Second thought is that once implemented the folder placement would to my mind then simplify the operation of creating separate pools and may actually lead me to do it, instead of just thinking about it  :)

 

I'm all for it!!!

 

.


  • Alex and Christopher (Drashna) like this

#6 Rychek

Rychek

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 20 September 2013 - 07:31 PM

At firtst I wasn't sure I would have any use for such a feature, but the more I think about it, the more I like the idea of having more control over what goes where.  It could be very useful as my children get old and skilled enough to use the server.  Bring on the progress!

 

Oh, yeah, and thanks so much for all your hard work Alex!  Drivepool and Scanner are awesome and the WS 2012 E integration in the last update felt like an early Christmas present. :)


  • Alex and Christopher (Drashna) like this

#7 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 23 September 2013 - 01:34 AM

Oh, yeah, and thanks so much for all your hard work Alex!  Drivepool and Scanner are awesome and the WS 2012 E integration in the last update felt like an early Christmas present. :)

 

Thank you that is very much appreciated :)


  • AMCross likes this

#8 Psilberman

Psilberman

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 September 2013 - 02:05 AM

This is something that I have hoped for, for a very long time. I really have never been keen on having things scattered around the pool. Especially with music... 10 album tracks scattered over 5 or 6 drives.. But then I'm probably a bit O.C.D.
.


I'm right there with you. My storage is mainly movie rips and i don't do file duplication due to the storage space required and I can just re-rip if needed. However, if a drive were to fail, i would have to re-rip most of my 1900 disks since the files are scattered everywhere.

I would love to be able to rip to the pool and have all the files for the rip go to one physical drive. This way recovery from a drive failure would be much easier.

#9 Shane

Shane

    Resident Guru

  • Moderators
  • 100 posts
  • LocationAustralia

Posted 26 September 2013 - 09:49 AM

I'd like the ability to say "keep together all files in any|each folder of folderpath" where "any" and "each" are modifiers affecting which files to keep together.

 

So given an example where folderpath was p:\media\sketches\* and I had files in p:\media\sketches\2012 and p:\media\sketches\2013

 

if the "any" modifier was selected 

then the files in sketches\2012 and sketches\2013 and any subfolders thereof would be kept all together

 

but if the "each" modifier was selected

and the files in sketches\2012 and any subfolders thereof would be kept together

and the files in sketches\2013 and any subfolders thereof would be kept together

yet sketches\2012 and sketches\2013 would be stored independently of each other.

 

Also, something along the lines of if folderpath ends in "folder\" it also applies to any files within that folder itself, while if it ends in "folder\*" it only applies to files within that folder's subfolders.

 

I hope this makes sense.


  • Christopher (Drashna) likes this

#10 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 27 September 2013 - 11:07 PM

Shane,

 

Very interesting.

 

I was thinking of doing it like this:

  • Internally the system would associate a standard path pattern with a set of pool parts.
  • For example:
    • \Sketches\2012\* ->Disk 0,1,2
    • \Sketches\2013\* -> Disk 2,3,4

And you would set up an unlimited amount of these patterns to configure any kind of folder placement strategy that you want. In addition, every rule would have a maximum fill limit (e.g. 90%), so that if you had to copy more files into a given folder than the set of disks (that store files for that folder) can contain, the rule would be violated and your "overflowing" files would be placed onto other disks. This is very similar to how our existing balancing system works.

 

Hmm... I'll think about if your suggestion can be implemented with my patterns scheme.



#11 Shane

Shane

    Resident Guru

  • Moderators
  • 100 posts
  • LocationAustralia

Posted 29 September 2013 - 08:47 AM

Thanks Alex. For another example, I know some people (including myself) like to store a backup of their DVD collections on their server for easy access; if they were keeping their library under p:\dvdlibrary\ with each DVD having its own folder, and wanted to ensure that all the VOB files for a given title were kept on the same disk (e.g. to avoid having a movie interrupted if the server had to wake the next drive out of sleep/standby):

 

with my suggestion they could do so with a single rule, "keep together all files in each folder of p:\dvdlibrary\*" and the pool would internally handle which disk each set of files were kept on

 

whereas with only basic path-to-disks matching, as soon as their library exceeded the size of a single disk in the pool they would have to micro-manage which title(s) were associated with which disk(s).

 

Of course, the underlying logic of my method might end up requiring DrivePool to dynamically generate some sort of path-to-disk ruleset to handle the internal housekeeping, but as a user I would prefer that not to be my problem (*grin*).



#12 SantiagoDraco

SantiagoDraco

    Member

  • Members
  • PipPip
  • 27 posts

Posted 22 October 2013 - 05:15 PM

Just saw this thread, should have read it sooner!    This sounds great and I posted another thread about creating a performance index using a new performance test that might be added to Scanner in order to determine real world drive performance.   That could be used to determine which drives to place which data.

 

My suggestion was more of an automatic thing (ie flag a folder as needing high/medium/low performance disc) using the same list we currently use for duplication... that said I can also see, after reading the thread, the benefit of manually associating folders with drives.   I like the Disney example.

 

One thing I think might need addressed is making sure that users are able to properly identify discs more specifically.   Ie not just showing drive letter/volume name but also the drive model, if possible.

 

I can't figure out why no applications really let you see the physical drives detailed information associated with the Windows drive letter/controller channel, etc.   Drives me batty sometimes trying to identify a disk :)

 

Anyway this is a great thread and I'm looking forward to some of the proposed enhancements.   I think Drive Pool is fantastic and hope to see it grow and really get noticed by the market.  These kinds of things will do that I think.



#13 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 23 October 2013 - 12:19 AM

Have you moused over the drives in DrivePool 2.x? It gives you the model number and serial number.
It also pulls the "bay and location" information from Scanner. So you can set those, and VERY easily identify the disk in DrivePool :)

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#14 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 16 November 2013 - 01:49 PM

So, i was just wondering if there is any further development on this? 

Im not meaning is it close to being implemened, more.. Is it still on the roadmap to happen?  :P



#15 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 16 November 2013 - 08:45 PM

Yes it is.

 

Right now, Alex is busy trying to add Reparse Point support before adding this.


  • daveyboy37 likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#16 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 17 November 2013 - 10:58 PM

Excellent news!! :)



#17 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 15 January 2014 - 10:51 PM

Was just wondering if this is this being worked on now or has Alex still got work going on with reparse points ? (I have googled reparse points and am still unsure what they do). But Kudos to Alex all the same for his tireless work.

I just bought 3 X 4TB drives and am on a mass data migration mission  :) to get rid of a couple of iffy (according to my scanner trial) 2TB drives. So my 9 pooled drives will soon be 8... But I really don't like having tracks off a single album scattered over 8 or 9 drives...It just doesn't make sense. 

So if it's still some way off then I will probably create a second pool with 2 of the drives and seeing as my music is one of the few folders I duplicate, it should in theory create 2 complete copies all neat and tidy. :lol:

That said, I'm still looking forward to being able to control folder placement from within drivepool itself.  :P



#18 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 16 January 2014 - 12:40 AM

http://blog.covecube...reparse-points/

 

Basically, we, we've added it, but we want to make sure that it's good and stable before rolling it out

Also, we've been working a lot and improving the backend of the site. Namely, the support system and related stuff, to make it easier for us to get to all of the tickets and deal with any issues that do arise.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#19 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 16 January 2014 - 01:18 AM

 

Basically, we, we've added it, but we want to make sure that it's good and stable before rolling it out

 

Just to clarify.. Are you talking about reparse points here, or the ability to control folder locations within the pool? 



#20 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,510 posts
  • LocationSan Diego, CA, USA

Posted 16 January 2014 - 05:20 AM

Reparse points. 

 

But the folder location thing is still on Alex's mind, and on his to-do list (for various reasons).

 

And I think I said it above, but the reparse points thing ended up being a LOT more tricky that Alex anticipated. This, and a couple of other things are the top of Alex's todo list (such as adding TrueCrypt support).  So, he definitely hasn't forgotten.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users