Best Way to Use Cloud Drive to Duplicate Data?

[Dizzy]

I just installed CloudDrive and got connected to BackBlaze.  My ultimate goal is to use Backblaze as a backup.  I had a drive recently crash HARD on me and I lost a ton of stuff.  I'm hoping to prevent that.

I have about 46TB of data that I need backed up.  I am not sure if I should create one big ass drive and just copy everything, and figure out a way to maintain it, and have CloudDrive basically as cold storage.  I noticed that the largest drive I could make is 10TB.  I have folders with 17TB and 18TB, so I can't even duplicate folders.

Do I create a bunch of 10TB drives, add them to the pool and duplicate?

What's the best way to use Backblaze to keep my data backed up?

 

Thanks!

[srcrist]

I don't think there is necessarily an objective "best" to provide you with. It sort of depends on your needs. You might consider using CloudDrive in a pool with your local storage using DrivePool or another similar tool and then running redundancy with Backblaze. That would back up your content AND repair it as necessary. Or you could simply set up rsync or a similar tool to copy data to the cloud periodically. 

In any case, the theoretical maximum size of a CloudDrive drive is 1PB. You can make a drive larger than 10TB, simply type in the size you want. Your volume size will, of course, be limited by the cluster size that you format it with. If you use 64KB clusters, you can make a single 256TB volume, though I would suggest you keep individual volumes lower than 60TB with NTFS, otherwise chkdsk will not work and there is no way to repair file system errors. I would say that if you need to backup 46TB, you could make a single 50TB volume on CloudDrive with 16KB clusters and that should suit you fine. 

[srcrist]

I should also point out that Backblaze's personal storage option provides unlimited backup space as long as you can use their in-house tool. That might be cheaper than using B2 cloud storage with CloudDrive, if backup is your only desire. 

[Dizzy]

Ah, I didn't realize that I could type in a size, I was just looking at the dropdown options.  Thanks for that... One thing solved.

After my wife lost a lot of HER data in the most recent drive failure, she agreed that it would be worth it to spend some money to ensure that we have things protected properly.

I like the idea of backup AND repair.  I have DrivePool, Scanner, and CloudDrive.  Could you walk me through how to get that set up?

I don't think I want to add the BackBlaze drives to the general pool since my data would be added to them in a way where I'd end up accessing that data from the Backblaze drives, and since downloading costs money, I really only want things downloaded if it's to restore/repair data.

As I write this and think about it, I'm leaning towards the personal.  I think I liked the idea of using CloudDrive, but it may not be the best in this case.  I have a few other ideas that CloudDrive will work nicely for though.

 

[srcrist]

I'm relatively certain that DrivePool can be configured in such a way that it only ever reads data off of your CloudDrive if your local storage is unavailable. You set it up by creating a pool with all of your local drives, adding your clouddrive, and enabling duplication only on the cloud drive. 

Good luck with whatever you decide!

[Bigsease30]

On 2/16/2019 at 6:30 PM, srcrist said:

I'm relatively certain that DrivePool can be configured in such a way that it only ever reads data off of your CloudDrive if your local storage is unavailable. You set it up by creating a pool with all of your local drives, adding your clouddrive, and enabling duplication only on the cloud drive. 

Good luck with whatever you decide!

Hello srcist,

I am in a similar boat as the OP, 24TB of location data stored in one pool using 13 spindle disks. Could you be slightly more detailed in your recent response of setup? I am curious to know if we are to add the cloud drive to the current data pool or make a new pool for it. If we have duplication already turned on for the local data pool, do we need to turn that off or is the program smart enough to re-dupe the data to the cloud drive automatically or will it copy both the original and duplicated files over?

[srcrist]

You can add the CloudDrive to your existing pool. Once you create the CloudDrive volume, it will show up in DrivePool as an option to add to the pool just like any other drive. Once you add the drive, you'll want to use the balancing options in DrivePool to adjust the settings you'd like to use. For example, you can adjust whether you want duplicated and unduplicated data on the CloudDrive volume, or if you just want it to hold duplicates exclusively. All of those sorts of settings are settings that you can adjust to make it work as you wish. Once the pool is added and you've set your preferred settings, it will begin to adjust the data on your drives in accordance with your preferences on the next balancing pass. 

DrivePool and its balancers provide a large number of settings that you can adjust as you need. You'll really just need to open up the balancer configurations and look at them in order to decide what suits your use case. Most likely you'll want to disable the placement of unduplicated data on the CloudDrive volume if you're only using it to back up a physical pool. You'll do that using a setting on the Drive Usage Limiter balancer, and then move that balancer over everything else so that it has the top priority. This means that all NEW data will be placed on your physical pool, and then copied to the CloudDrive volume as a duplicate slowly, over time.