Jump to content
  • 0

The right way to mirror only to cloud?


Thronic

Question

Seems tickets are on a backlog, so I'll try the forums as well. I've setting up a new 24 drive server and looking first at stablebit as a solution to pool my storage since I already have multiple licenses and somewhat experience with it, just not with clouddrive.

I've done the following.

Pool A:
Physical drives only.

Pool B:
Cloud drives only.

Pool C:
A and B with 2x duplication. Drive Usage set to B only having duplicate files. This let me hold off turning on 2x duplication until I have prepared local data (I have everything in the cloud right now), so the server doesn't download and upload at the same time. Pool A has default settings. Pool B have turned off balancing, I don't want it to start downloading and uploading just to balance drives in the cloud. It's enough with the overfill prevention. 

 

My thought process is that if a local drive goes bad or need replacement, users of Pool C will be slowed down but still have access to data via the cloud redundancy. And when I replace a drive, the duplication on Pool C will download needed files to it again. Is read striping needed for users of Pool C to always prioritize Pool A resources first?

This almost seems too good to be true, can I really expect it to do what I want? I have 16TB download as well as Pool B having double upload (2x cloud duplication for extra integrity) before I can really test it. Just wanted to see if there are any negative experiences with this before continuing. My backup plan is to just install a GNU/Linux distro instead as a KVM hypervisor and create a ZFS or MDADM pool of mirrors (for ease of expansion) with a dataset passed to a Windows Server 2019 VM on a SSD (backed up live via active blockcommit) and hope GPU passthrough really works. But it surely wouldn't be as simple ... I know there is unraid too, but it doesn't even support SMB3 dialect out of the box and I'm hesitant to the automatic management of all the open source software stacks involved.. Heard of freezes and lockups etc.. Dunno about it.

Regardless, any of the backup solutions would simply use rclone sync as I've used so far for user data backups. Which would not provide live redundancy like hierarchical pools, so I'd loose local space for parity based storage or mirroring. I wont have to loose any local storage capacity at all, if this actually works as expected.

Link to comment
Share on other sites

4 answers to this question

Recommended Posts

  • 0

Sorry, I haven't used CloudDrive nearly enough to be able to tell you whether your plan is actually sound (I use DrivePool and Scanner a lot more) though that may change in the near-ish future.

"Drive Usage set to B only having duplicate files." - Have you tested that this works with a pair of local pools? If so, it should also work with a local+cloud pool pairing.

"Is read striping needed for users of Pool C to always prioritize Pool A resources first?" - According to the manual the answer is no; regardless of whether Read Striping is checked it will not stripe reads between local and cloud pools, instead it tries to only read from the cloud pool if it can't read from the local pool. However, I have noted forum posts about issues with getting DP to prioritise correctly, and I don't know if those issues have been fixed since those posts as I haven't dug through the CloudDrive changelog recently. https://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance Options.

The rest of your plan seems sound.

Link to comment
Share on other sites

  • 0
On 2/16/2021 at 1:13 AM, Thronic said:

Drive Usage set to B only having duplicate files

There isn't a way to do this, since StableBit DrivePool treats both copies of the files the same. Eg, there is no "original" and "duplicate".  

But enabling duplication on the top level pool, and enabling read striping means that it will prefer the local pool over the cloud pool. 

On 2/16/2021 at 1:13 AM, Thronic said:

Pool B have turned off balancing, I don't want it to start downloading and uploading just to balance drives in the cloud. It's enough with the overfill prevention. 

No reason to turn off balancing then.  Balancing ONLY kicks in on edge cases, by default. https://stablebit.com/Support/DrivePool/2.X/Manual?Section=Balancing Plug-ins#Default Plug-ins

 

 

Link to comment
Share on other sites

  • 0
11 minutes ago, Christopher (Drashna) said:

StableBit DrivePool treats both copies of the files the same. Eg, there is no "original" and "duplicate".

What I meant with that option is this:

Since I haven't activated duplication yet, the files are unduplicated. And since the cloud pool is set to only duplicated, they should land on the local pool first. And when I activate duplication, the cloud pool will accept the duplicates. Working as intended so far, but I'm still hesitant about this whole setup. Like I'm missing something I'm not realizing, but will hit me on the ass later.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...