Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Drivepool duplicating to only Clouddrive..


  • Please log in to reply
18 replies to this topic

#1 renxwar

renxwar

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 09 June 2017 - 01:45 PM

Hello, I've created a CloudDrive connecting to Google Drive, then added the created disk. I would like to duplicate the existing data to just the CloudDrive (essentially creating a 1:1 physical to cloud). I enabled Duplication, then under "Balancing..." and "Drive Usage Limiter", I deselected Duplicated on all phyiscal (leaving Unduplicated checked, and on Google Drive I checked Duplicated and unchecked Unduplicated.

 

When I began duplicating, it's spreading it evenly across all my drives, not just the CloudDrive. Is there a setting I missed to restrict to just the CloudDrive?


  • Antoineki likes this

#2 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 09 June 2017 - 02:09 PM

AFAIK, the Duplicated setting in the Drive Usage Limiter does not ensure *one* of the duplicates to go to such a drive and the other copy to the ones having Unduplicated data. As both are duplicates, none of the two copies are unduplicated and would end up at the unduplicated ones.

 

Not sure which version of DP you run but the later betas allow Pool hierarchies. I use that currently. So I have 2 Pools, each consisting of actual HDDs and these are unduplicated. Then I create a 3rd Pool (The Mother Of All Pools as I call it) that consists of the 2 existing Pools and this one is duplicated. This does exactly what you are looking for (in my case, ensuring one copy on one seperate set of HDDs and the other on another seperate set of HDDs which, in your case, might be the Google Drive). I *think* in your case you could do:

1. Create a Pool with actual HDDs, unduplicated

2. Create a Pool with the Pool from (1) above and CloudDrive, duplicated.

 

DP will then hold one copy in Pool (1) and one on CloudDrive.



#3 renxwar

renxwar

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 09 June 2017 - 05:54 PM

I'm on the latest beta version of DrivePool and CloudDrive.

 

So basically what you're saying is ultimately having 2 different pools (1: Physical / 2: CloudDrive), then making a pool of those 2?



#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 09 June 2017 - 06:22 PM

There is latest public version, and latest internal version. 

2.2.0.651 and 2.2.0.754.   

It's the second that has the hierarchical pooling.  

 

 

So basically what you're saying is ultimately having 2 different pools (1: Physical / 2: CloudDrive), then making a pool of those 2?

 

Yes, this would be the best way. As you could ensure that one copy is local, and one is on the CloudDrive drive(s).

 

In fact, this is exactly why the hierarchical pooling was developed.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 08 August 2017 - 05:54 AM

So, just to be clear on this, suppose I have all my data in Pool 1, with various folders duplicated across multiple HDDs.  I now create Pool 2 from a cloud drive.  Pool 2 is completely empty and it will remain unduplicated.  I now create the master pool, Pool 3 and add Pools 1 and 2 to it.  Pool 3 will show all the data in Pool 1 as "other".  But if I enable pool duplication on Pool 3 (for now the only one I can enable since there are no folders), then *everything* in Pool 1 will be duplicated to the cloud drive in Pool 2?

 

I did this, but obviously "other" data in Pool 1 will not be duplicated.  I don't quite understand how this can be made to duplicate across HDDs and also to the cloud drive inside Pool 2.

 

Thanks.



#6 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 378 posts

Posted 08 August 2017 - 04:37 PM

I think the issue is that Pool 1 can have data that is not in Pool 3. What you want to do, but test with a few files first, is *move* files from Pool 1 to Pool 3.

 

I think you may have two poolpart folders on HDD 1 now. First one for Pool 1 and within that folder another for Pool 3. You might manage quicker by moving files from one poolpart (being Pool 1) to the other (being Pool 3 on HDD 1). Not sure how that works if you have shares etc, those may need re-establishing.

 

The thing is, you *can* store files in Pool 3 *and* Pool 1 side by side and each with their own rule. I'd advise aganst it but it can be done.



#7 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 08 August 2017 - 11:27 PM

I'm still not understanding this properly.  To Pool 3, any data stored in other pools (directly) simply look like blobs that are unusable space.  If I drop something into Pool 3 then it will be duplicated in Pools 1 and Pools 2 but is inaccessible from within those pools.  Because of this, it also will not be duplicated on multiple HDDs *within* Pool 1.  This does not serve the original purpose, which is to keep something safe by dropping it in Pool 1 (comprised of local HDDs) and make certain it will *always* be duplicated at least once in Pool 2 (containing just the single cloud drive).  As has been done in other forum posts, I will outline my (what I think are fairly simple) requirements here and I hope someone can tell me how to achieve this using hierarchical pools (or some other method that's not "breaking the rules" or inadvisable).  I'm willing to move stuff around:

 

  1. I currently have a pool, D: comprising several HDDs with several folders duplicated (not the whole pool).
  2. I have a Clouddrive, B: on Box
  3. I would like all existing and future files in D: to always be duplicated to B: at least once, regardless of whether they are already duplicated in D:

That's it.  I've seen people in other posts mess around with drive usage limiting and balancing, and I know I can always just "backup" to B: using another program, but it seems I should be able to do this via the stellar combination of these two products.  I hope @Christopher can chime in soon on this.



#8 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 08 August 2017 - 11:53 PM

So, just to be clear on this, suppose I have all my data in Pool 1, with various folders duplicated across multiple HDDs.  I now create Pool 2 from a cloud drive.  Pool 2 is completely empty and it will remain unduplicated.  I now create the master pool, Pool 3 and add Pools 1 and 2 to it.  Pool 3 will show all the data in Pool 1 as "other".  But if I enable pool duplication on Pool 3 (for now the only one I can enable since there are no folders), then *everything* in Pool 1 will be duplicated to the cloud drive in Pool 2?

 

I did this, but obviously "other" data in Pool 1 will not be duplicated.  I don't quite understand how this can be made to duplicate across HDDs and also to the cloud drive inside Pool 2.

 

Thanks.

 

Mostly.

 

The data on the Pool will not be added to the "master pool".  You'd need to seed that data, manually.

 

(eg, move it into the newly created, hidden "PoolPart" folder on the pool, and remeasure the master (top level) pool.

 

Once you've done that, it will show up in the pool, and show up as properly measured.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#9 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 09 August 2017 - 07:28 AM

But Drashna, the problem with this is that once the files are moved off Pool 1, they are no longer protected by local duplicates, only the cloud duplicate.  Neither will new files added to the master pool be protected by local duplicates.  This is actually hurting my brain!



#10 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 10 August 2017 - 11:51 PM

Yeah, this can get complicated fast.

 

 

The duplication status is maintained independently for each pool

 

That means that you can have manage the duplication level for both the master pool, and for the local pool.

 

 

For the master pool, you'd want to enable "Pool File duplication", and not set the folder duplication. 

This makes sure that one copy is on the CloudDrive, and one is on the local pool.

 

 

For the local pool, you can enable duplication there, as well.  If you want everything duplicated locally (so that there are two copies on the local disks), you can do that.   If you only want one copy local, you can disable duplication for that folder or the whole pool.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#11 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 11 August 2017 - 12:41 AM

Hi Chris,

 

Thanks for that answer and I don't want to be a PITA but I'm still not seeing a method for what I'm trying to achieve.  Specifically, using DrivePool and CloudDrive, how do I ensure that at least one duplicate of specified folders exists on local drives and another duplicate exists on my cloud drive?



#12 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 11 August 2017 - 02:32 AM

The above method is the best way to do this.   It will ensure that it happens, and it will not cause issues (other than being more complex).

 

Otherwise it is this:

http://community.cov...bit-clouddrive/

But this will cause the pool condition to be "bad" and will want to constant balance your data.

 

 

 

 

But to clarify, you want one copy local, and one copy in the cloud, and that this is guaranteed, correct? 

 

But you don't care where the local copy is. 

 

 

 

If you only want one copy locally.... then enable duplication on the master pool, and don't enable it on the "local" secondary pool.

 

If you want two copies locally, then enable duplication on both.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#13 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 11 August 2017 - 05:31 AM

OK, thanks again Chris for your patience (I'm starting to sound like what we call in Yiddish a "pechech").  There still seems to be an impedance mismatch here.  Please correct my understanding as I walk through what I put together from the very beginning of this thread and what I'm currently seeing (I know I wrote this in a previous post but I'd like you to refer specifically to what's laid out here):

 

  1. I have existing files in a DrivePool, D: comprising only local HDDs.  This pool has 2x duplication on several folders, but let's just assume for simplicity that the entire pool is duplicated.  So, there are 2 copies of every file somewhere on the local HDDs.
  2. I've created a CloudDrive, B:
  3. I've created a new DrivePool, E:
  4. I've added D: and B: to the "master" pool E: and set 2x duplication on the entire pool.
  5. At this point, looking at E: in the GUI shows me that 4.57TB is Other and 7.78TB is unusable for duplication.  Specifics aside, I assume the 4.57TB is the data in D:
  6. If I do nothing else from this point on, the 4.57TB that's located inside D: will never be duplicated to B: because E: doesn't see that data as something that needs to be duplicated.  I assume this is correct based on your comment above about "seeding" the master pool.
  7. If I instead move a folder out of D: and into the hidden PoolPart folder for the master pool and re-measure, then that folder will exist locally on D: (on some HDD that's part of that pool) and also be duplicated on B:.  However, it will no longer be duplicated on multiple local HDDs because the D: pool doesn't know about it (i.e., it doesn't show up in a listing of the D: drive).
  8. The same applies to any future data that I write to E:.  In fact, if I look at the folder tree in the Folder Duplication panel for D:, DrivePool (E:\) shows up as disabled.
  9. In addition, any future data that I write to D: will not be duplicated on B:

So I find that there is no situation in which files stored in any pool will be guaranteed to have 2+ copies locally and 1+ copies in the cloud.

 

Please tell me which statements above are incorrect.



#14 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 11 August 2017 - 10:15 AM

No worries.  This is a complicated config, and I'm not doing the best to explain it (it's been a long few weeks, sory).
 
 
6 and on out are incorrect.
 
Specifically, on the D: pool, when you add that to the "master" pool, you'll see a hidden "PoolPart" folder on that pool.  
So if you look at the underlying disk, you'd see "X:\PoolPart.xxxx\PoolPart.xyyy".  And on the D:\ pool, you'd just see D:\PoolPart.xyyy.
 
Just move the contents of D:\ into the PoolPart.xyyy folder.  
 
 

E: (Master)
├───B: (CloudDrive)
├───D: (Local Pool)
│   ├───X: 
│   ├───Y: 
│   ├───Z: 

The content that you add to "E:\" will reside on both B:\ and D:\.  And the content that resides on D:\ will reside on the remaining drives.

 

 

However, you do need to move the contents on D:\ into the D:\PoolPart.xyyy folder for it to show up in E:\ (and you'd need to remeasure the pool, so it shows it properly and balances and duplicates it properly).

 

 

And yes, any data that you write to D:\ would outside of the PoolPart.xyyy folder would not be duplicated onto B:\.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#15 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 12 August 2017 - 01:16 AM

Thanks again Chris for making things much clearer!  This is my last message for a while because I don't want to tax you good folks at CoveCube any longer.

 

With a set of simple tests I have now confirmed at least part of what I thought might be true.  It's unfortunate because it means that I can't really use the DP/CD combo to do exactly what I want (and I don't think it's an unreasonable use case).

 

I've moved a x2 folder mytest from D:\ to D:\PoolPart.xxyy and remeasured E:. Initially, it remained on HDD1 and HDD2 and I could see it being duplicated to B:.  But then, something got to rebalancing (probably the D: pool) and I watched as mytest was quickly removed from HDD1.  At this point, mytest only resides on HDD2 and on B:\.  This clearly breaks my requirement that I have at least 2 copies on local drives.  I should mention that all my balancing rules are defaults.

 

What's more, dropping files onto the E: pool does not duplicate them on local hard drives either.  I can confirm all of this both from the File Placement tab in the balancing panel for D: and by confirming where these files exist on HDD.

 

Unless you have any further suggestions, I'll just have to "backup" to my cloud drive using a traditional program.



#16 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 12 August 2017 - 02:25 AM

You're very welcome.  Ironically, the formatting is due to a nice thread on reddit, mentioning the "tree" command, that outputs in this format.  I saw it, and was like "that's EXACTLY what I need for this thread".  (no, seriously, that's literally what popped into my head)

 

And please don't worry about taxing/bugging/harassing/dragging me through the mud.  I really don't mind.   It's literally what I'm here for!   So, if you have a question, don't hesitate to ask!

 

 

As for the issues, I'll have to look into this directly and verify.  If it is doing what you described consistently.... that's a bug, and needs to be fixed.

 

And to make sure, do you see the PoolPart.xyyy folder on the "local" pool, when checking duplication?

Also, try setting the master pool to x3 duplication and see if that works (or even lets you)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#17 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 13 August 2017 - 03:21 AM

Okay, it looks like this behavior was "by design", and mostly because it is very complex.

 

That said, Alex has added this in the latest version.  

 

The caveats here are:

You need to enable the "show hidden folders" option in the Folder Duplication window to see the Pool data.

You can only toggle the status for the pool storage, in general.  Meaning this is an "all or nothing" setting.  you can't have some of the pool duplicated and some of it not.  It's not great, but adding detailed options is not going to be easy.  

 

So for the time being, it's possible, just not with any fine control.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#18 chiamarc

chiamarc

    Member

  • Members
  • PipPip
  • 14 posts

Posted 13 August 2017 - 09:10 PM

Brilliant!  Tested and satisfied!  This gets me *much* closer to my goal and I can now drop files into E: and be sure that they are duplicated locally and into the cloud!  This will suffice for the time being as I can always keep important stuff on E: and not so important stuff on D:.

 

Thanks again for being so quick to respond (and change code)!


  • Christopher (Drashna) likes this

#19 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,361 posts
  • LocationSan Diego, CA, USA

Posted 13 August 2017 - 11:34 PM

I'm glad to hear it, and you're very welcome.

 

And Alex (the Developer) has been in active development of DrivePool, so really, this was good timing.  Otherwise, it can take a while to get to requests/issues like this.

 

Either way, glad to hear it!


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users