Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

StableBit DrivePool - Controlling Folder Placement


  • Please log in to reply
53 replies to this topic

#21 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 14 February 2014 - 02:26 PM

The only concern I would have is how DP would deal with removal of drives. Say you have a pool of 4 drives, 2x duplication and videos go to drive 3 and 4. Removing drive 3 would then result in, uhm, what?

 

I think this is why my experience with DP 2.x on removing/adding is somewhat dissapointing. I have a 2x2TB pool, 2x duplication. Had to remove 1. I'm sure I did not use Best Practice but it is unclear to me what the best way is to deal with this. So I am sort of an idiot with weird self-induced problems but I can see something similar becoming an issue with folder placement restrictions / directives.

 

It does appear, to me, somewhat contradictory to the fundamental notion of a Pool (or _my_ notion of a Pool) and I would not use it and would fear for user-induced data loss. Setting up seperate Pools seems a way more consisten way to go.

 

Anyway, do what you want. I got DP&S and I'm very happy with what I got!


  • vapedrib likes this

#22 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 16 February 2014 - 06:21 AM

I would assume that the Balancer would then use ANY available drive, as according the other balancers. As per normal.

 

As for... well, self induced problems, you're definitely not alone. :P

But as for the pool, If all the files were duplciated, then you could just remove a disk and then "remove" the missing disk from the pool. It would be unable to duplicate, but it would be "okay". Once you've attached another disk, it would want to duplicate to that "new" disk.

 

As for best practice, "Remove" link is the best method. That, or if you have the disks and space, and uptime is critical, then use "File Placement Limiter" to clear off the disk.

 

(it may be that I'm tired, but) I'm not sure what you mean by this:

"It does appear, to me, somewhat contradictory to the fundamental notion of a Pool (or _my_ notion of a Pool) and I would not use it and would fear for user-induced data loss. Setting up separate Pools seems a way more consisten way to go."

 

 

And as for "do what you want", we are very driven by user feedback. It's important to us. Even if we never use the feature, there are those that may.  Your opinions do matter to us!


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#23 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 17 February 2014 - 09:39 AM

Well, the way I see it, a Pool is an abstraction layer over physical HDs (or actually, the poolpart folders on the related HDs) that represents or behaves like another HD. That DP does all kinds of stuff within (like duplication on different physical HDs, read-striping, allocation algorythms etc) is great but it is indifferent to the "kind of file" or "name of folder" etc. Folder Placement would, I think. break some sort of clean distinction between the physical HDs and the abstraction layer and I would fear that administration might become complicated (both for a user as for DP itself) and potentially contradictory.

 

I would really wonder what the net advantage would be over, for instance, defining various Pools. So, IF you forced a folder "Videos" to only store on HD 2, and HD 2 becomes full, what shoudl DP do? Spill over any excess to another HD? Then you've failed to meet the objective of having all "Videos" on HD2. Move other data from HD2 to other HDs to make room for "Videos"? Meanwhile meeting read-striping/performance and duplication optimisation/constraints? And what if HD2 is really full with only "Videos"? Perhaps you'd _want_  a "Disk Full" message, which a seperate Pool would actually get you....

 

It's just fishy to me. In part perhaps because my Pool is very small, I use a 2-disk, 2x duplication so I essentially have a very easy, smart and recoverable RAID-1 minus all the hassle of RAID. Folder Placement would be silly in my case. Anyway, do what you want ;) .

 

Edit: maybe someday I'll get me a real testing box. "Removing" a drive just seemed to take ages in my set-up. Perhaps I messed up because I essentially wanted to remove + de-duplicate as I was going to a 1-drive setup temporarily. Adding a drive appeared to take ages as well, perhaps as balancing and re-duplication is not a noce combination. Might be specific for a 2-disk 2x-duplication setup. Found that actually clearing the Pool (moving folders off-Pool), killing Pool, changing 1 HD, format + re-create Pool, moving back to Pool for me worked way way faster.



#24 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 17 February 2014 - 04:50 PM


It's just fishy to me. In part perhaps because my Pool is very small, I use a 2-disk, 2x duplication so I essentially have a very easy, smart and recoverable RAID-1 minus all the hassle of RAID. Folder Placement would be silly in my case. Anyway, do what you want ;) .

 

 

 

 

And that's just it. With your set up it would be pointless...but

For those of us with more drives (12 in my case) with a huge collection of music , it isn't ideal to have tracks off a single album scattered (theoretically)  over all those drives. The only option is to create a second pool to give any sort of control. 

Taking on board Drashnas comments, I'm guessing the feature is likely to be in the shape of a balancer, then as with all the other balancers they can all be toggled on or off. Nothing is forced on you. There will be nothing restrictive if you simply choose not to use it.

Sorry but it just comes across that you feel it's a lame feature purely on the basis that it will be of no use to you personally. I say again....don't use it. 

In the meantime... I am being FORCED to use a separate pool to achieve what I need Drivepool to do. It's just nice to have options. 



#25 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 17 February 2014 - 05:46 PM

Indeed, I do not know your situation, I don't know why it would be suboptimal to have tracks off a single album scattered and it's certainly not for me to say whether it'd be a good feature for you or in general or not. I just hope it won't harm stability and won't cause issues due to administrative errors.

 

Sorry for being critical, I would only hope that if my comments are not complete nonsense, they might at least contribute to a better product.



#26 Shane

Shane

    Resident Guru

  • Moderators
  • 100 posts
  • LocationAustralia

Posted 18 February 2014 - 12:07 AM

Long story short: folder placement should come "off" by default, and it should be completely up to you whether you want the extra administrative responsibility of using it to get that "extra mile" out of your pools.

 

Long story long: folder placement is primarily a tool for (1) pools with large number of drives and/or (2) pools that are not using full duplication.

 

1. As the number of drives grow, so does the impact of physically scattered files on power consumption (inefficiently waking multiple drives from standy) and latency (waiting while those drives come out of standby) and bandwidth (if multiple users are accessing different areas of the pool, physical scattering increases the odds of any two users competing for the same physical disks) from negligible to significant.

 

2. In dealing with storage, there is a mantra ignored at peril: "RAID Is Not Backup". Since drives are not free, sometimes we have to choose between having enough drives for a fully duplicated pool and enough drives for the backup(s) - and for vital data you should always choose the latter. However, some backup software may lack certain features, such that performing a partial restore of physically scattered files is awkward (or to be more blunt, a right pain in the butt).

 

Also, 3. Properly implemented, it provides an efficient alternative to the current awkward workaround of splitting files into multiple pools (which has its own drawbacks).



#27 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 18 February 2014 - 06:35 AM

There is also a missing point. The main reason behind the balancers in the first place is so we can be OCD about the file placement. :) 

 

But yeah, it's not for everyone. And there are some that definitely want this feature. Much like the already optional balancers. And yes, they would be able to be "toggled" and the priority rearranged as well.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#28 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 18 February 2014 - 06:48 AM

Power consumption, point 1 of Shane, is certainly one I had not realised. The OCD argument of course wholly is sufficient, I just do not happen to be OCD re file placement: I hear ya!



#29 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 18 February 2014 - 06:59 AM

Actually, all three of Shane's points are very good ones.

 

And yeah, OCD is a big factor in why I love DrivePool. The very "hands off" approach is great for those that are OCD. :)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#30 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 18 February 2014 - 10:46 AM

I'm not denying that, just had not thought of that one entirely/at all.



#31 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 18 February 2014 - 10:58 AM

 

 

And yeah, OCD is a big factor in why I love DrivePool. The very "hands off" approach is great for those that are OCD. :)

 
And I have enough OCD to share with everyone else.  :)
My motherboard has 8 built in Sata ports (6 sata III and 2 Sata II). Also a 2 port Sata II card and a 4 port Sata I card. I'm using 13 of the 14 available ports. 
 
For a long long time I have hoped to be able to replace my 2TB drives with 3 TB drives but It never quite happened until now. Fortunately it has taken me so long to upgrade that price wise, I have been able to skip the 3TB and upgrade straight to 4TB. Two of them are installed and another three should be on there way to me today or tomorrow.  
So I have around 15TB of data on my system and its all precious to me. I don't have the storage overhead to duplicate everything and probably 12TB of my data is MKV movies. The vast majority of that is not that important. If a hard drive dies I will have no problem re-downloading Terminator 2, Independence Day, Fight Club etc...
But what about the much rarer movies such as an early 80's Australian movie 720p that someone ripped off the TV. I'm pretty sure I will never find that again. 
So for me the choice is either migrate all my data onto a drive that must be duplicated (that should keep me busy for a good few days), or tell Drivepool to duplicate that movie.. Its a no-brainer and for me at least THE one thing that I badly need from Drivepool. Then with some organisation afterwards I could probably upgrade my 2TB server backup drive to a 4TB and even include the above mentioned difficult to get stuff in Server backup as well as having them duplicated. Media and OCD  heaven for me. :)
When the other 4TB drives arrive, the two cheapo sata cards will get ripped from the system and replaced with a 2 or 4 port Sata III card when needed. This will give me scope for another 8-16TB of space that I will probably struggle to fill until I'm 6 foot under.
 
Please. please, please Alex.. Bring it on soon!!! :D
 
 
Edit... I'm really getting myself confused now... I do know know that I can duplicate any folder already but the ability to control placement would be optimal for me. I just really want to know that all my duplicated music & movies reside on drives 1 & 4 etc... 


#32 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 18 February 2014 - 12:00 PM

Nevertheless, I would think that any OCD person would like to get a clear signal if and when, e.g.,

- Drives 1 & 4 are (or the smaller of the is) are full and files are spilling over onto other drives.

- Drive 1 or 4 is failing and/or is removed which would violate the folder placement directive etc.

 

But yeah, I'm getting the point of folder placement now.



#33 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 18 February 2014 - 12:54 PM

Nevertheless, I would think that any OCD person would like to get a clear signal if and when, e.g.,

- Drives 1 & 4 are (or the smaller of the is) are full and files are spilling over onto other drives.

- Drive 1 or 4 is failing and/or is removed which would violate the folder placement directive etc.

 

But yeah, I'm getting the point of folder placement now.

 

 

Again this depends on organisation and on an individuals system and how they use it. I personally would never let the drives get to the position they would overflow. In my case Server 2012r2 and from past experience WHS2011 gives plenty of notice of such issues and thoughtful initial setup of the existing balancers will simply take care of the rest. The server backup on the operating systems mentioned are a pita to initially setup. You cannot just select which folders to back up. You have to browse to each individual serverpoolpart folder on each drive and select the relevant folders. With a 2 or 3 drive system that's no big deal but with 10-15 drives that's pretty time consuming. If music is scattered over a dozen drives and you miss selecting it on one of the drives backup will fail or at least be incomplete. Then it's back to all those drives to see what was missed. Mutiply that by the 10 or 15 different server folders and its a nightmare. :o If my music is on two drives, then that's  two drives i need to check that it's being backed up properly.  

 

Having replaced the numerous 2TB drives with the 4TB ones I personally will have plenty of scope to add another 4TB drive as and when needed. The folder placement will simply be another option for those who feel they need it. 

 

Don't get me wrong. I am currently using two pools just to make sure all my flac files are duplicated but to be honest for me at least it just negates the benefit of having a pool in the first place. 

 

Anyway i'm glad you are starting to realise why limiting folder placement will be a massive benefit to some (if not all) of us. In years to come as your system grows you may also come to realise just how much of a benefit. Though hopefully you will never get as OCD as Drashna and myself. :lol:

 

Finally with regard to the drive failing, I actually see limiting placement as a good thing. My personal plan is to have all my "must not lose because i probably cannot replace data" duplicated over two drives. If one of those drives fails. My full data is still on the other drive. A much better scenario than a random drive failing and losing a couple of tracks off most if not every single album.


  • Christopher (Drashna) likes this

#34 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 18 February 2014 - 07:32 PM

Hahaha, I will _never_ (oops, shouldn't use that word) use folder placement. I understand the backup issue though, in fact, that is _why_ I only have two 2TB drives in my Pool: I get the stability of RAID 1 and can still backup everything using WHS backupsoftware by backing up one of the two drives. A 3 drive-pool would no longer allow me to do that, unless I were to use folder placement. I'll simply not let it grow any larger, LOL.

 

I believe WS2012E already allows larger server backups but this all is a bridge I'll cross when I find it.



#35 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 18 February 2014 - 11:26 PM

@daveyboy37: Very, very well put!

 

@Umfriend: :)

And to each their own. But that is why we have a list of "extra" balancers, the framework for balancing, API for developing your own (if you're inclined), etc. If you're noticing a theme, "choice" is a big consideration for DrivePool, and we hope that it is appreciated. :)

 

And yes, Server 2012 allows for large backups. Specifically, the backup engine has always (well, the "Windows Backup" feature, not ntbackup) used VHDs for storing the backups. If you look it up, VHD's were limited to 2TB size, until Server 2012, which introduced the VHDX format which allows for a 64TiB per VHDX file. :)


  • daveyboy37 likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#36 Swap_File

Swap_File

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 February 2014 - 04:41 PM

I think I would find a "Try to keep files in directory X on disk Y" or "Try to keep directories together (In general or below file or directory size Z)" option extremely useful.

 

Hypothetical Situation:  I have a bunch of ripped CDs stored on my home server.  It's a directory (named Music) of many smaller directories (named Album Title) made up of smaller files (Each Song).  DrivePool spreads these songs evenly between my two drives.   If one drive dies, I could quite possibly lose half my songs from every album, meaning I would need to re-rip all of my CDs to restore my collection.  

 

Ideally I would have easily accessible backups, or have DrivePool set to keep multiple copies of everything, but that does not always happen.  I don't mind taking a calculated risk for some low priority files, but I would still prefer to minimize the amount of work it takes to recover from a failure.

 

In my view, for sets of files that depend on each other, my exposure to damage to the set goes up the more dispersed the set is.  Meaning that for non-duplicated sets of files, DrivePool could be riskier than non-pool storage.  In the above example, a single drive failure could damage all of my albums.  If my music storage was clumped by album, worst case a single drive failure would cause the complete loss of half my albums.

 

Both failures involve data loss, but depending on the situation, one can be much more painful than the other to recover from.  Even with backups.

 

Related question: 

 

I'm relatively new to using DrivePool.  Does DrivePool have a way of keeping track of what files are on what drive, so in the event of a failure, I could figure out which sets of files (or albums) were affected?  Or do I need to be keeping track of that on my own?  For the most part its easy to tell if a data set is whole, but some things are harder than others.  Using the music example, It's easy to tell if a track in the middle of an Album is missing, but less easy to tell if the last track disappears.



#37 Shane

Shane

    Resident Guru

  • Moderators
  • 100 posts
  • LocationAustralia

Posted 24 February 2014 - 01:37 AM

Indeed, the scenario you describe is one of the reasons why the mantra "RAID is not backup" exists.

 

Q. Does DrivePool have a way of keeping track of what files are on what drive, so in the event of a failure, I could figure out which sets of files (or albums) were affected?  Or do I need to be keeping track of that on my own?

 

A. No - it does not maintain such a list. Yes - you would need your own method of keeping track.



#38 Swap_File

Swap_File

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 February 2014 - 08:48 PM

Thanks for the clarification.  I realize RAID is not backup, but in the event catastrophic RAID failure, I at least know everything in the array is gone and that I would need to restore everything from backup.

 

It seems like with DrivePool during a drive failure I could lose some data and not be sure what data was lost or what folders were affected (unless I keep track of it externally, or compare all of my data against a backup).  The problem would be exacerbated by how DrivePool keeps sets of files spread across multiple drives.

 

The Ordered File Placement plugin is a good start at fixing that, but a bit more control would be greatly appreciated!

 

I'll have to look into what indexing programs are available to keep track of my files.



#39 daveyboy37

daveyboy37

    Advanced Member

  • Members
  • PipPipPip
  • 41 posts

Posted 24 February 2014 - 11:21 PM

Well hopefully , the ability to control folder placement Isn't too far away. :P



#40 Shockwave818

Shockwave818

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 27 February 2014 - 10:35 PM

Hi all, made an account expressly to respond to this thread.

I totally would love the ability to control folder placement.  I've been looking for a feature like this forever!

 

Scenario:

Force Action movie folder to physical Disk 1

Force Scifi movie folder to physical Disk 2

Physical disk 1 and 2 appear as a single disk within explorer

 

- If physical disk 1 fails, I only lose my Action movies.

- If phyiscal disk 2 fails, I only lose Scifi movies.

- If disk 1 gets full and I try to write to Action movies, I get "disk full message"

- If disk 2 gets full and I try to write to Scifi movies, I get "disk full message"






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users