Drive pool backups/Keeping files in folders together

[APoolTodayKeepsTheNasAway]

Hi, I apologize if this is too general or not the right place to post this, but I havent been able to find an answer thus far, so Im posting this in the hopes I can get the answers without testing myself.

I currently have quite a few hard drives in a jbod collection.

What I hope to do is have one pool of 15TB of in use drives, combined with a total of 28TB in new storage where I can have a pool of 21tb in always on drives backing up periodically to a pool of 22tb TBD drives which would be either for file duplication in a raid 1 fashion or backup. 

What Im unsure of is how I can accomplish this with Drive pool. I imagine I would create the first always on pool, then create the second one, simply turning off and disconnecting all the drives it uses when finished backing up then plugging them back in to refresh said backup.

What Im hoping would be the case is that in the event of a failure, I would be able to see what files and folders where on the failed drives and restore them from the backup. hopefully automatically without disturbing anything else.

----------------------------------------------------------------------------------------------------

Separately to that as it seems I can't test it on some usb drives I have lying around (can usb flash drives work?!), I would love to know how granular the pooling system is.

Say for instance, I have 1000 folders differently named and filled with multiple files, can I then set Drive pool to only move around folders with certain attributes keeping all their child files and perhaps folders within?

The way that would practically work, is that upon placing a new file into a folder,  instead of rebalancing the location of the file based on size, the whole folder would be moved.

Reading through the documentation in the form of the FAQ I cant seem to find a way to accomplish this, which would be great for restoring from backups in the case of a drive failure or accidental deletion.

File Placement options seems very close to but just short of this unless Im missing a way to configure it to act this way.

----------------------------------------------------------------------------------------------------

Additional information: I plan to be filling some sort of external multi bay usb enclosure to hold the additional drives for my windows 10 computer using the Stablebit DrivePool and Scanner to manage all of the drives.

 

 

[Christopher (Drashna)]

This gets asked fairly frequently.  There isn't a good way to ensure this.  

The Ordered File Placement balancer plugin makes a "best effort" but doesn't ensure it.
The File Placement rules can be used to do this, but requires a lot of micromanaging.

 

And the current balancing system isn't designed to do this... and would basically require a complete rewrite to accomplish.

[APoolTodayKeepsTheNasAway]

2 hours ago, Christopher (Drashna) said:

This gets asked fairly frequently.  There isn't a good way to ensure this.  

The Ordered File Placement balancer plugin makes a "best effort" but doesn't ensure it.
The File Placement rules can be used to do this, but requires a lot of micromanaging.

 

And the current balancing system isn't designed to do this... and would basically require a complete rewrite to accomplish.

Thanks for the response.

I have no idea how big a change it would be but, something similar to file placement rules, like folder placement rules would mostly accomplish what Im looking for with 2 pieces of information. Like for example a rule that goes if number of files (direct child(not counting files in sub folders)) > number of folders (direct child(not counting folders in sub-folders)) ; keep together. This would allow far less micromanagement but keep most folders recognizable.

Also, about the backup part, would an offline backup  using drivepool be possible (safe), as in can I take a pool of say 4 drives, create a pool on them, to backup another pool, and take them offline to update them periodically. Short of that would you happen to know any backup software that could keep daily backups of filenames/folder structure to restore missing files?

[B00ze]

Hi A Pool A Day Keeps The Doctor Away.

There was a big discussion about this, see here: File Placement Based On Folder (towards the end I added a few replies on the problems I'd run into if I was to code this myself...)

As for backup, just so you know, if 1 drive is missing from a pool, you can no longer write to that pool until you re-balance (i.e. remove the drive from the UI and see what happens.) So at least you'd have to make sure you always reconnect all the USB drives. Christopher can say for sure about what would happen when 2 pools are related by duplication and then 1 pool goes offline, I've never tried that before.

Regards,

[APoolTodayKeepsTheNasAway]

33 minutes ago, B00ze said:

Hi A Pool A Day Keeps The Doctor Away.

There was a big discussion about this, see here: File Placement Based On Folder (towards the end I added a few replies on the problems I'd run into if I was to code this myself...)

That would work too I think, That requires a lot more micromanagement though than  if #offolders>#offiles;keep together however.

Quote

As for backup, just so you know, if 1 drive is missing from a pool, you can no longer write to that pool until you re-balance (i.e. remove the drive from the UI and see what happens.) So at least you'd have to make sure you always reconnect all the USB drives. Christopher can say for sure about what would happen when 2 pools are related by duplication and then 1 pool goes offline, I've never tried that before.

Regards,

What I mean is 2 separate pools. The plan is to have one pool offline most of the time as a backup for the other pool so pool 1 would have 4 drives, and occasionally I would connect pool 2 drives to sync pool 1 contents. Im just wondering if there are any inherent problems with that (mainly to do with taking a pool offline.

[B00ze]

Hi. If you just mount the missing pool now and again, and use copy/robocopy/syncToy/whatever to synchronize the 2 pools, then I don't see any issues. Just remember that you always have to reConnect ALL the drives from Pool 2, or the pool will be read-only. I wish someone else who'd tried that before would jump in; DrivePool is not supposed to have any issues with the scenario you describe, so that's what I'm saying, but it's always better if someone answers that has actually done it before (taking drives offline/online repeatedly.) I've done something similar, and DrivePool detected my pool just fine, but I've only done it a couple times, I've not tried this on a regular basis.

Regards,

[APoolTodayKeepsTheNasAway]

59 minutes ago, B00ze said:

,but it's always better if someone answers that has actually done it before (taking drives offline/online repeatedly.) I've done something similar, and DrivePool detected my pool just fine, but I've only done it a couple times, I've not tried this on a regular basis.

Regards,

Yea, I was really  hoping @Christopher (Drashna) could confirm this would work for the long term, because while Im grateful for the replies of other users, only developers/associates really will be able to give a firm/definitive answer.

 

What Im kind of afraid of is Il start using this for backups, then an update will come which makes the inactive pool unable to be read or used correctly in Drivepool which would severely hurt its usefulness as a backup given a key focus of the backup would be retaining organization.

[Christopher (Drashna)]

Yeah, adding this would be a huge change, and potentially a big performance impact.  Also, how far "up and down" the directory tree would be kept together, etc. 

And checking would be done in the kernel, and that would add significant wait time for each new file.  

 

So, unfortunately, it's not really practical with how the balancing system is configured, and would require a massive rewrite. 

And for the file placement stuff, that was a huge addition (IIRC), but it used the existing code extensively.  Folder based would basically require brand new code for the entire system. 

 

(at least that's my guess, as I don't do the coding part)