Jump to content
  • 0

Drivepool duplicating to only Clouddrive..


renxwar

Question

Hello, I've created a CloudDrive connecting to Google Drive, then added the created disk. I would like to duplicate the existing data to just the CloudDrive (essentially creating a 1:1 physical to cloud). I enabled Duplication, then under "Balancing..." and "Drive Usage Limiter", I deselected Duplicated on all phyiscal (leaving Unduplicated checked, and on Google Drive I checked Duplicated and unchecked Unduplicated.

 

When I began duplicating, it's spreading it evenly across all my drives, not just the CloudDrive. Is there a setting I missed to restrict to just the CloudDrive?

Link to comment
Share on other sites

Recommended Posts

  • 0

AFAIK, the Duplicated setting in the Drive Usage Limiter does not ensure *one* of the duplicates to go to such a drive and the other copy to the ones having Unduplicated data. As both are duplicates, none of the two copies are unduplicated and would end up at the unduplicated ones.

 

Not sure which version of DP you run but the later betas allow Pool hierarchies. I use that currently. So I have 2 Pools, each consisting of actual HDDs and these are unduplicated. Then I create a 3rd Pool (The Mother Of All Pools as I call it) that consists of the 2 existing Pools and this one is duplicated. This does exactly what you are looking for (in my case, ensuring one copy on one seperate set of HDDs and the other on another seperate set of HDDs which, in your case, might be the Google Drive). I *think* in your case you could do:

1. Create a Pool with actual HDDs, unduplicated

2. Create a Pool with the Pool from (1) above and CloudDrive, duplicated.

 

DP will then hold one copy in Pool (1) and one on CloudDrive.

Link to comment
Share on other sites

  • 0

There is latest public version, and latest internal version. 

2.2.0.651 and 2.2.0.754.   

It's the second that has the hierarchical pooling.  

 

 

So basically what you're saying is ultimately having 2 different pools (1: Physical / 2: CloudDrive), then making a pool of those 2?

 

Yes, this would be the best way. As you could ensure that one copy is local, and one is on the CloudDrive drive(s).

 

In fact, this is exactly why the hierarchical pooling was developed.

Link to comment
Share on other sites

  • 0

So, just to be clear on this, suppose I have all my data in Pool 1, with various folders duplicated across multiple HDDs.  I now create Pool 2 from a cloud drive.  Pool 2 is completely empty and it will remain unduplicated.  I now create the master pool, Pool 3 and add Pools 1 and 2 to it.  Pool 3 will show all the data in Pool 1 as "other".  But if I enable pool duplication on Pool 3 (for now the only one I can enable since there are no folders), then *everything* in Pool 1 will be duplicated to the cloud drive in Pool 2?

 

I did this, but obviously "other" data in Pool 1 will not be duplicated.  I don't quite understand how this can be made to duplicate across HDDs and also to the cloud drive inside Pool 2.

 

Thanks.

Link to comment
Share on other sites

  • 0

I think the issue is that Pool 1 can have data that is not in Pool 3. What you want to do, but test with a few files first, is *move* files from Pool 1 to Pool 3.

 

I think you may have two poolpart folders on HDD 1 now. First one for Pool 1 and within that folder another for Pool 3. You might manage quicker by moving files from one poolpart (being Pool 1) to the other (being Pool 3 on HDD 1). Not sure how that works if you have shares etc, those may need re-establishing.

 

The thing is, you *can* store files in Pool 3 *and* Pool 1 side by side and each with their own rule. I'd advise aganst it but it can be done.

Link to comment
Share on other sites

  • 0

I'm still not understanding this properly.  To Pool 3, any data stored in other pools (directly) simply look like blobs that are unusable space.  If I drop something into Pool 3 then it will be duplicated in Pools 1 and Pools 2 but is inaccessible from within those pools.  Because of this, it also will not be duplicated on multiple HDDs *within* Pool 1.  This does not serve the original purpose, which is to keep something safe by dropping it in Pool 1 (comprised of local HDDs) and make certain it will *always* be duplicated at least once in Pool 2 (containing just the single cloud drive).  As has been done in other forum posts, I will outline my (what I think are fairly simple) requirements here and I hope someone can tell me how to achieve this using hierarchical pools (or some other method that's not "breaking the rules" or inadvisable).  I'm willing to move stuff around:

 

  1. I currently have a pool, D: comprising several HDDs with several folders duplicated (not the whole pool).
  2. I have a Clouddrive, B: on Box
  3. I would like all existing and future files in D: to always be duplicated to B: at least once, regardless of whether they are already duplicated in D:

That's it.  I've seen people in other posts mess around with drive usage limiting and balancing, and I know I can always just "backup" to B: using another program, but it seems I should be able to do this via the stellar combination of these two products.  I hope @Christopher can chime in soon on this.

Link to comment
Share on other sites

  • 0

So, just to be clear on this, suppose I have all my data in Pool 1, with various folders duplicated across multiple HDDs.  I now create Pool 2 from a cloud drive.  Pool 2 is completely empty and it will remain unduplicated.  I now create the master pool, Pool 3 and add Pools 1 and 2 to it.  Pool 3 will show all the data in Pool 1 as "other".  But if I enable pool duplication on Pool 3 (for now the only one I can enable since there are no folders), then *everything* in Pool 1 will be duplicated to the cloud drive in Pool 2?

 

I did this, but obviously "other" data in Pool 1 will not be duplicated.  I don't quite understand how this can be made to duplicate across HDDs and also to the cloud drive inside Pool 2.

 

Thanks.

 

Mostly.

 

The data on the Pool will not be added to the "master pool".  You'd need to seed that data, manually.

 

(eg, move it into the newly created, hidden "PoolPart" folder on the pool, and remeasure the master (top level) pool.

 

Once you've done that, it will show up in the pool, and show up as properly measured.

Link to comment
Share on other sites

  • 0

But Drashna, the problem with this is that once the files are moved off Pool 1, they are no longer protected by local duplicates, only the cloud duplicate.  Neither will new files added to the master pool be protected by local duplicates.  This is actually hurting my brain!

Link to comment
Share on other sites

  • 0

Yeah, this can get complicated fast.

 

 

The duplication status is maintained independently for each pool

 

That means that you can have manage the duplication level for both the master pool, and for the local pool.

 

 

For the master pool, you'd want to enable "Pool File duplication", and not set the folder duplication. 

This makes sure that one copy is on the CloudDrive, and one is on the local pool.

 

 

For the local pool, you can enable duplication there, as well.  If you want everything duplicated locally (so that there are two copies on the local disks), you can do that.   If you only want one copy local, you can disable duplication for that folder or the whole pool.

Link to comment
Share on other sites

  • 0

Hi Chris,

 

Thanks for that answer and I don't want to be a PITA but I'm still not seeing a method for what I'm trying to achieve.  Specifically, using DrivePool and CloudDrive, how do I ensure that at least one duplicate of specified folders exists on local drives and another duplicate exists on my cloud drive?

Link to comment
Share on other sites

  • 0

The above method is the best way to do this.   It will ensure that it happens, and it will not cause issues (other than being more complex).

 

Otherwise it is this:

http://community.covecube.com/index.php?/topic/1226-how-can-i-use-stablebit-drivepool-with-stablebit-clouddrive/

But this will cause the pool condition to be "bad" and will want to constant balance your data.

 

 

 

 

But to clarify, you want one copy local, and one copy in the cloud, and that this is guaranteed, correct? 

 

But you don't care where the local copy is. 

 

 

 

If you only want one copy locally.... then enable duplication on the master pool, and don't enable it on the "local" secondary pool.

 

If you want two copies locally, then enable duplication on both.

Link to comment
Share on other sites

  • 0

OK, thanks again Chris for your patience (I'm starting to sound like what we call in Yiddish a "pechech").  There still seems to be an impedance mismatch here.  Please correct my understanding as I walk through what I put together from the very beginning of this thread and what I'm currently seeing (I know I wrote this in a previous post but I'd like you to refer specifically to what's laid out here):

 

  1. I have existing files in a DrivePool, D: comprising only local HDDs.  This pool has 2x duplication on several folders, but let's just assume for simplicity that the entire pool is duplicated.  So, there are 2 copies of every file somewhere on the local HDDs.
  2. I've created a CloudDrive, B:
  3. I've created a new DrivePool, E:
  4. I've added D: and B: to the "master" pool E: and set 2x duplication on the entire pool.
  5. At this point, looking at E: in the GUI shows me that 4.57TB is Other and 7.78TB is unusable for duplication.  Specifics aside, I assume the 4.57TB is the data in D:
  6. If I do nothing else from this point on, the 4.57TB that's located inside D: will never be duplicated to B: because E: doesn't see that data as something that needs to be duplicated.  I assume this is correct based on your comment above about "seeding" the master pool.
  7. If I instead move a folder out of D: and into the hidden PoolPart folder for the master pool and re-measure, then that folder will exist locally on D: (on some HDD that's part of that pool) and also be duplicated on B:.  However, it will no longer be duplicated on multiple local HDDs because the D: pool doesn't know about it (i.e., it doesn't show up in a listing of the D: drive).
  8. The same applies to any future data that I write to E:.  In fact, if I look at the folder tree in the Folder Duplication panel for D:, DrivePool (E:\) shows up as disabled.
  9. In addition, any future data that I write to D: will not be duplicated on B:

So I find that there is no situation in which files stored in any pool will be guaranteed to have 2+ copies locally and 1+ copies in the cloud.

 

Please tell me which statements above are incorrect.

Link to comment
Share on other sites

  • 0

No worries.  This is a complicated config, and I'm not doing the best to explain it (it's been a long few weeks, sory).
 
 
6 and on out are incorrect.
 
Specifically, on the D: pool, when you add that to the "master" pool, you'll see a hidden "PoolPart" folder on that pool.  
So if you look at the underlying disk, you'd see "X:\PoolPart.xxxx\PoolPart.xyyy".  And on the D:\ pool, you'd just see D:\PoolPart.xyyy.
 
Just move the contents of D:\ into the PoolPart.xyyy folder.  
 
 

E: (Master)
├───B: (CloudDrive)
├───D: (Local Pool)
│   ├───X: 
│   ├───Y: 
│   ├───Z: 

The content that you add to "E:\" will reside on both B:\ and D:\.  And the content that resides on D:\ will reside on the remaining drives.

 

 

However, you do need to move the contents on D:\ into the D:\PoolPart.xyyy folder for it to show up in E:\ (and you'd need to remeasure the pool, so it shows it properly and balances and duplicates it properly).

 

 

And yes, any data that you write to D:\ would outside of the PoolPart.xyyy folder would not be duplicated onto B:\.

Link to comment
Share on other sites

  • 0

Thanks again Chris for making things much clearer!  This is my last message for a while because I don't want to tax you good folks at CoveCube any longer.

 

With a set of simple tests I have now confirmed at least part of what I thought might be true.  It's unfortunate because it means that I can't really use the DP/CD combo to do exactly what I want (and I don't think it's an unreasonable use case).

 

I've moved a x2 folder mytest from D:\ to D:\PoolPart.xxyy and remeasured E:. Initially, it remained on HDD1 and HDD2 and I could see it being duplicated to B:.  But then, something got to rebalancing (probably the D: pool) and I watched as mytest was quickly removed from HDD1.  At this point, mytest only resides on HDD2 and on B:\.  This clearly breaks my requirement that I have at least 2 copies on local drives.  I should mention that all my balancing rules are defaults.

 

What's more, dropping files onto the E: pool does not duplicate them on local hard drives either.  I can confirm all of this both from the File Placement tab in the balancing panel for D: and by confirming where these files exist on HDD.

 

Unless you have any further suggestions, I'll just have to "backup" to my cloud drive using a traditional program.

Link to comment
Share on other sites

  • 0

You're very welcome.  Ironically, the formatting is due to a nice thread on reddit, mentioning the "tree" command, that outputs in this format.  I saw it, and was like "that's EXACTLY what I need for this thread".  (no, seriously, that's literally what popped into my head)

 

And please don't worry about taxing/bugging/harassing/dragging me through the mud.  I really don't mind.   It's literally what I'm here for!   So, if you have a question, don't hesitate to ask!

 

 

As for the issues, I'll have to look into this directly and verify.  If it is doing what you described consistently.... that's a bug, and needs to be fixed.

 

And to make sure, do you see the PoolPart.xyyy folder on the "local" pool, when checking duplication?

Also, try setting the master pool to x3 duplication and see if that works (or even lets you)

Link to comment
Share on other sites

  • 0

Okay, it looks like this behavior was "by design", and mostly because it is very complex.

 

That said, Alex has added this in the latest version.  

 

The caveats here are:

You need to enable the "show hidden folders" option in the Folder Duplication window to see the Pool data.

You can only toggle the status for the pool storage, in general.  Meaning this is an "all or nothing" setting.  you can't have some of the pool duplicated and some of it not.  It's not great, but adding detailed options is not going to be easy.  

 

So for the time being, it's possible, just not with any fine control.

Link to comment
Share on other sites

  • 0

Brilliant!  Tested and satisfied!  This gets me *much* closer to my goal and I can now drop files into E: and be sure that they are duplicated locally and into the cloud!  This will suffice for the time being as I can always keep important stuff on E: and not so important stuff on D:.

 

Thanks again for being so quick to respond (and change code)!

Link to comment
Share on other sites

  • 0

Sorry for opening up an old post :unsure: But I'd like to double check before I do something "stupid"

I've created the cloudDrive and that works as expected but now I want to backup my entire Drivepool to the cloud using that clouddrive. AND at the same time keep some folders duplicated locally (Folder Duplication) Is this the way to go?:

1. Got my normal DrivePool (A:)

2. Created my CloudDrive (D:)

3. Add both D: and A: to a new Pool (E:)

4. Enable pool file duplication (E:)

5. Move all my Files from A: to A:\PoolPart.XXXYYY

6. Go into the normal DrivePool (A:) enable folder duplication on A:\HybridPool\FolderName

 

And then make sure all new Files are placed on (E:) ?

And if this is the way to go, can I remove the drive letters?

Link to comment
Share on other sites

  • 0

Mostly correct, yep.  :) 

If A: is already a pool, step 5 is redundant.  The files would already be inside the hidden Poolpart-xxxxx folder(s) for it's drives.  To add to that pool, you'd just move files into the existing drive letter assigned to it (A: I assume).

You can -manually- move files that are already on a volume (but outside the Poolpart-xxxxx folder) to a Poolpart-xxxxx folder in the Pool, but you'll want to do a re-measure on the Pool after that so it's aware of the new additions.

You're correct in creating a hierarchical Pool (E:) and adding A: and D: to it.  If you're duplicating the entire pool there's no need to enable folder duplication, just turn on Pool duplication on E: and set it to 2x.  Not sure what folders you want to duplicate locally, but I'm assuming they're inside Pool A:, so you could easily enable folder/file duplication for them from Pool A's management interface.  Duplication on a pool requires >1 volume in the pool, so A: will have to have multiple volumes assigned.

Any files that you put on E: will get auto-duplicated to A: and D:.  You'll want to do a re-measure and then kick off a manual re-balance, so that your Clouddrive is fully populated with files from A:.  Since you're using a Clouddrive for one of the volumes, you can turn off real-time duplication (a performance setting).

You can remove drive letters after the hierarchical pool is created, sure.  And if you want you can use Folder Mount Points to keep them easily available (like from a C: root folder).  You do that in the same place in Windows Disk Management where you'd normally assign it a letter.  Just make an empty folder there for it first, then go assign the drive to the folder in Disk Management.

Link to comment
Share on other sites

  • 0

@Jaga Step 5 would still be required, if you want to move the data into the E:\ pool, though. 

 

Also, don't use A:\ or B:\.  Windows is hard coded to treating these drives as floppy drives, and may cause odd behavior. For instance, the system will ping these disks every 5 seconds or so, IIRC. 

Link to comment
Share on other sites

  • 0

@anderssonoscar, this has been mentioned elsewhere but I think it's important enough to repeat:  technically, the pool that you have "backed up" to the cloud is *replicated* to the cloud.

There is a distinct difference between backup and replication that is lost on some people.  If you make a change to something in your pool (including deletion), that change is propagated to the cloud drive in a (potentially very) small window of time.  This window is of course dependent on the number of files and amount of data being replicated.  My point is that it is not a backup, or if it is, it's a very weak one!  A backup implies that you can always restore files that were lost, deleted, or otherwise changed, intentionally or unintentionally.  StableBit replication is designed to always keep a specified number of copies of something in your locations of choice so that at least one copy of that something will be likely to survive in the event of corruption or destruction in one or more of the locations.  It cannot and should not be used reliably as a backup.

Personally, I do this:  I replicate important stuff locally with DP but I use a backup program to backup to my CD.  Unimportant stuff (i.e., easily reproduced or re-obtained) may simply be stored on my CD without replication or replicated locally.

Link to comment
Share on other sites

  • 0

 

56 minutes ago, Christopher (Drashna) said:

Also, don't use A:\ or B:\.  Windows is hard coded to treating these drives as floppy drives, and may cause odd behavior. For instance, the system will ping these disks every 5 seconds or so, IIRC. 

Sadly true - 25+ years later and the underlying drive system still wants to treat these differently.

Link to comment
Share on other sites

  • 0
1 hour ago, Christopher (Drashna) said:

@Jaga Step 5 would still be required, if you want to move the data into the E:\ pool, though. 

True, if it was a single drive pool and the files were outside on the same disk.  I may have made too many assumptions on the pool(s) architecture.  :rolleyes:

Link to comment
Share on other sites

  • 0
18 hours ago, Jaga said:

Sadly true - 25+ years later and the underlying drive system still wants to treat these differently.

Sadly, it's worse that that.  I am pretty sure it's maintained for backwards compatibility for whatever poor soul (sys admin, most likely) is still forced to use floppy drives. 

18 hours ago, Jaga said:

True, if it was a single drive pool and the files were outside on the same disk.  I may have made too many assumptions on the pool(s) architecture.  :rolleyes:

No worries. Seeding a pool from a pool is one of the more trippy things in StableBit DrivePool. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...