3,2,1 rule using DrivePool and CloudDrive

[collapsed]

My apologies if this has been discussed before...

 

I want to implement the 3,2,1 rule (3 copies, 2 local, one remote/cloud) using DrivePool and CloudDrive. For important files, I would like 2 copies of files stored locally (spanned at least two disks, which I'm already doing) with a third copy stored in the cloud via a CloudDrive disk added to the pool. How can I use the balancers to enforce that a (third) copy is always made specifically to the CloudDrive disk, and not just spanned to a random third disk in the local pool? 

 

Beyond that, for less important data where a single local copy is made, how can I enforce that a (second) copy is always made specifically to the CloudDrive disk, and not just spanned to a random second disk in the local pool? 

 

Might be a nice optimizer option to add to enhance tiered storage, like the SSD optimizer. Maybe SSD, HDD, Archive, Cloud, etc.

[Christopher (Drashna)]

Right now, there isn't a good way to accomplish this.

 

We already do have a "duplication grouping" feature on the to-do list to facilitate this. But that's not here yet, and may be a complicated bit of code.

 

 

 

In the meanwhile, you could use the file placement rules and set it only on the cloud drive, or use the "Disk Usage Balancer" to set only the cloud drive to have duplicated data.

 

These will both accomplish what you want, but they will cause the pool condition to be "unhealthy" as it's violating the rules/balancing options for the other drives. 

[jammin]

 

I want to implement the 3,2,1 rule (3 copies, 2 local, one remote/cloud) using DrivePool and CloudDrive. For important files, I would like 2 copies of files stored locally (spanned at least two disks, which I'm already doing) with a third copy stored in the cloud via a CloudDrive disk added to the pool.

 

Also please be aware that this stragegy alone does not protect you against an unnoticed accidental deletion or corruption of files (either from user error, malware, software bug etc).  Any deletion or corruption will be propogated to all 3 copies virtually instantly.

 

I'm designing a strategy at the moment and am trying to take this into account.  I'm thinking it is better to have a backup process using backup software that can restore state to a given time.  Back this up to a local drive (either in the pool or seperate) and then mirror it offsite.  I'm thinking of using Backup For Workgroups for this as it has the mirroring built in.

[Christopher (Drashna)]

Also please be aware that this stragegy alone does not protect you against an unnoticed accidental deletion or corruption of files (either from user error, malware, software bug etc).  Any deletion or corruption will be propogated to all 3 copies virtually instantly.

 

I'm designing a strategy at the moment and am trying to take this into account.  I'm thinking it is better to have a backup process using backup software that can restore state to a given time.  Back this up to a local drive (either in the pool or seperate) and then mirror it offsite.  I'm thinking of using Backup For Workgroups for this as it has the mirroring built in.

Absolutely correct. 

 

It's a decent start, though.  

 

Though, what would be better is using something to copy the files to a cloud drive on a schedule. (and one that doesn't delete destination files). 

[flylowlikerylo]

 

My apologies if this has been discussed before...

 

I want to implement the 3,2,1 rule (3 copies, 2 local, one remote/cloud) using DrivePool and CloudDrive. For important files, I would like 2 copies of files stored locally (spanned at least two disks, which I'm already doing) with a third copy stored in the cloud via a CloudDrive disk added to the pool. How can I use the balancers to enforce that a (third) copy is always made specifically to the CloudDrive disk, and not just spanned to a random third disk in the local pool? 

 

Beyond that, for less important data where a single local copy is made, how can I enforce that a (second) copy is always made specifically to the CloudDrive disk, and not just spanned to a random second disk in the local pool? 

 

Might be a nice optimizer option to add to enhance tiered storage, like the SSD optimizer. Maybe SSD, HDD, Archive, Cloud, etc.

 

 

So, I might be misunderstanding what you are attempting to do but from what it sounds like, I was under the impression I had done this exact thing (except 4,3,1 if you will).  In drivepool, I selected the desired folders to be duplicated 4 times under file protection.  Then under Balancing > File Placement I selected those folders to be placed on 3 local drives and 1 cloud drive by selecting the drives in the "file placement options" section.  Now I was under the impression that this would result in one copy of the folders being placed on each of the four designated drives.  After balancing, when I look at the folders it shows them being duplicated across the four drives (4,3,1) but from reading your question and the subsequent responses am I to understand that the 4 copies of the folder are not duplicated across the 4 drives?  Or was it that you were attempting to specifically use a balancer to accomplish this?

[collapsed]

Also please be aware that this stragegy alone does not protect you against an unnoticed accidental deletion or corruption of files (either from user error, malware, software bug etc).  Any deletion or corruption will be propogated to all 3 copies virtually instantly.

 

I'm designing a strategy at the moment and am trying to take this into account.  I'm thinking it is better to have a backup process using backup software that can restore state to a given time.  Back this up to a local drive (either in the pool or seperate) and then mirror it offsite.  I'm thinking of using Backup For Workgroups for this as it has the mirroring built in.

Oh yeah, not a problem. Thanks for looking out though.

Local shadow copies, differential backups, offline copies, etc... It's all good.

 

So, I might be misunderstanding what you are attempting to do but from what it sounds like, I was under the impression I had done this exact thing (except 4,3,1 if you will).  In drivepool, I selected the desired folders to be duplicated 4 times under file protection.  Then under Balancing > File Placement I selected those folders to be placed on 3 local drives and 1 cloud drive by selecting the drives in the "file placement options" section.  Now I was under the impression that this would result in one copy of the folders being placed on each of the four designated drives.  After balancing, when I look at the folders it shows them being duplicated across the four drives (4,3,1) but from reading your question and the subsequent responses am I to understand that the 4 copies of the folder are not duplicated across the 4 drives?  Or was it that you were attempting to specifically use a balancer to accomplish this?

As Drashna stated earlier, this is an option but depending on your configuration it may cause the pool condition to be "unhealthy" as it's violating the rules/balancing options for the other drives. 

 

 

So, as a workaround, I kicked around the idea of doing differential backups to the cloud, using CloudDrive. Unfortunately, the latest version is on shaky 'experimental' ground with Amazon. Hopefully this gets sorted out soon.

 

Oh, and FYI; I was thinking about using my license of Acronis for differentials (since I already had a license). For some reason, it'll only let me get a sector-level backup of my DrivePool, NOT differential. Nuts! :/

So that means using Acronis for diff backups is out.... and so is Amazon.... Double Nuts.... this is turning into a pair of fat nuts.

 

Ah well, it's still BETA. All in due time.

[flylowlikerylo]

As Drashna stated earlier, this is an option but depending on your configuration it may cause the pool condition to be "unhealthy" as it's violating the rules/balancing options for the other drives. 

 

 

So, as a workaround, I kicked around the idea of doing differential backups to the cloud, using CloudDrive. Unfortunately, the latest version is on shaky 'experimental' ground with Amazon. Hopefully this gets sorted out soon.

 

Oh, and FYI; I was thinking about using my license of Acronis for differentials (since I already had a license). For some reason, it'll only let me get a sector-level backup of my DrivePool, NOT differential. Nuts! :/

So that means using Acronis for diff backups is out.... and so is Amazon.... Double Nuts.... this is turning into a pair of fat nuts.

 

Ah well, it's still BETA. All in due time.

 

Yeah, I guess I'm not clear on what would cause the pool condition to be "unhealthy" or why doing this would violate balancing rules if you configured them correctly.  The only balancers I use are StableBit Scanner, Prevent Drive Overfill, and Duplication Space Optimizer in that priority.  I find it pretty straight forward to just set the folder duplication and then direct the pool to place folders only on the drives I want.  I realize my pool may be relatively simple as it consists of only 4 local drives (single volume on each) and one cloud drive - but this has been a straightforward way to accomplish a 4,3,1 setup (or so I thought).  

 

In any case I had to uninstall CloudDrive for now as it has caused my server to become unstable (cloud drive keeps dropping out of the drive pool and causing issues with the server being unable to restart and preventing access to the server remotely).  I couldn't even remove the cloud drive from the drive pool without crashing the DrivePool UI.  After uninstalling CloudDrive the server is stable again.  At no point did DrivePool indicate the pool was "unhealthy" (other than the cloud drive dropping out).

 

I was using Dropbox as the provider and CloudDrive would perpetually say that it needed to be reauthorized so that might have been at the root of the issue.  There definitely appeared to be non-network related communication issues at play. 

 

I also Attempted using Amazon as the provider but that was a nonstarter.  If Google Drive becomes an option at some point I will reinstall and give that a try. 

[Christopher (Drashna)]

So, I might be misunderstanding what you are attempting to do but from what it sounds like, I was under the impression I had done this exact thing (except 4,3,1 if you will).  In drivepool, I selected the desired folders to be duplicated 4 times under file protection.  Then under Balancing > File Placement I selected those folders to be placed on 3 local drives and 1 cloud drive by selecting the drives in the "file placement options" section.  Now I was under the impression that this would result in one copy of the folders being placed on each of the four designated drives.  After balancing, when I look at the folders it shows them being duplicated across the four drives (4,3,1) but from reading your question and the subsequent responses am I to understand that the 4 copies of the folder are not duplicated across the 4 drives?  Or was it that you were attempting to specifically use a balancer to accomplish this?

 

If you have the folder(s) set to appear on 4 different drives, and they have room, then there is absolutely no issue.

 

However,  if your creating a roll for a number of drives taht is LESS than the duplication, then it will cause the pool to become "unhealthy".

 

For instance, if you want two copies, one on the cloudDrive and once ... anywhere else, right now, you'd need to create a rule (or use a balancer) to limit the placement to that drive and only that drive.  

Since it has more copies than valid locations, that causes the unhealthy status. 

 

If you have as many copies as available drives, there is no issue.