Cloud Drive + G Suite = Backup disk

[Bigsease30]

Hello All,

I have been using Drivepool for a long time now and love it. I am now looking to further protect my files so I purchased Cloud drive and G Suite Business today. I currently have 34.6TB of total space on my server with 8.71TB free. Duplication is currently turned on using (2X). I have connected my Google Drive to the cloud drive and created a 50TB drive. Assuming that this maybe to much now after figuring out that I am only actually using 13TB of space if I were to remove duplication from my current setup. I configured the download to use 10 threads and the upload to use 5 with an upload limit of 70mbps to avoid being banned.  To further add fuel to the fire, most of my data is movies and TV shows that currently are being utilized from Plex media player.

What I would like to achieve is to have a local copy of my files and have them dupe to my cloud drive. I would like for Plex to only use the local files and not attempt to re-download any files. Doing this would allow me to reclaim my duplicated space on my local server. I have tried to figure it out myself but have been struggling the last few hours. I don't want to accidentally delete my pool and lose everything. Any help would be appreciated. I have included screen shots of my drive pool and Cloud Drive. Please let me know if you need additional information to assist me.

Thanks in advance.

[srcrist]

EDIT: I think this is wrong, actually. Let me experiment a bit. 

[Bigsease30]

15 minutes ago, srcrist said:

OK, so this relates to the other thread. So if you ONLY want copies on the CloudDrive volume, you'll simply use the Drive Usage Limiter to enable only duplicated files on the CloudDrive volume, and then *disable* duplicated files on the local volumes. Then with 2x duplication, all of the second copies will be stored on the CloudDrive volume and only the CloudDrive volume. 

DrivePool is smart enough to use local storage for retrieval if it is available, over the CloudDrive volume. 

It will not mess up your pool. It will simply start migrating the data once a balancing pass begins. This process will, obviously, take a long time with that amount of data. Likely months. So be aware of that. 

Thanks for the reply. I found the location of the Drive usage limiter but how do I select the Cloud Drive as an option? It is not listed in the list of drives (see new screenshot). Under my drivepool, (see photo above) it doesn't list it to be able to be added to the original pool either.

[srcrist]

The CloudDrive volume will show up in DrivePool as a drive to be added to the pool once you've created it. Are you saying you've created a volume and it's still not showing up?

I think my previous idea was wrong regardless. Trying to play with some options now. 

[Bigsease30]

Yes, that is correct. I have already created the Cloud Drive as shown in the screenshot above. It doesnt show as an option to be added to the current pool.

[srcrist]

I think what we actually need to do is create a NEW pool using your old pool and the CloudDrive volume. Disable duplication on your old pool entirely, and then enable 2x duplication in the new pool. That will duplicate the entire pool to the CloudDrive volume. Give that a test and see how it works. Though this doesn't solve the problem of your CloudDrive volume not showing up in DrivePool. I'm not sure why that would be. 

It might be because the pool is ReFS. Is the CloudDrive NTFS? I'm not sure if DrivePool can merge them. 

If you create a NEW pool, can you add a CloudDrive volume to that, in any case?

[srcrist]

It looks like mixed pools ARE possible, but not recommended. So that might be why you're not seeing the option to add the drive. See here: https://community.covecube.com/index.php?/topic/2832-can-i-mix-ntfs-and-refs-drives-in-my-pool-drivepool-version-22-beta/

[Bigsease30]

Ok, when I click create pool, this is what I see. How do i remove the drives from my current pool without losing data?

[srcrist]

We don't want to remove them at all. We're creating a hierarchical pooling scheme. We are using your old pool to pool all of your local drives together, and then a new pool to duplicate the data between the pool and your CloudDrive. So leave your old pool as it is, but disable the 2x duplication on that pool. Then add your D: and your Z: to this new pool, and enable 2x duplication on that. 

[Bigsease30]

OK. Just so I understand this correctly. I will end up with 3 pools?

1) Z:\DrivePool (Current - All local disks)
2) D:\Google Drive (New - Cloud Drive)
3) E:\Combined (Newer - Includes DrivePool and Google Drive)

[srcrist]

Well the D: is not a pool. It's just a CloudDrive volume. You'll have two pools, which will be nested. 

Though, thinking about it, we'll need to also move all your existing data to the new poolpart folder that will be created on the old pool in order to make your existing data accessible to the NEW pool. So that may take a minute. 

Let me do this in a test with a smaller pool to make sure nothing crazy happens. Give me a minute. 

[srcrist]

OK. So, here is the process as far as I can tell:

Create a new pool. Add your Z: and D: to the new pool. Disable duplication on your Z: pool entirely. Use the drive usage limiter on the NEW pool to set the CloudDrive volume to ONLY accept duplicated data. Move ALL existing content on Z: to the new poolpart folder on Z:, which shouldn't take long since DrivePool is smart enough to do it on the same drive. Once all of the data is accessible from the new pool, simply enable 2x duplication on the new pool, and it should slowly migrate all of the duplicates to D:.

Does that make sense?  

Once you're done with all of that, you can use Z: for the new pool and everything should be identical as far as the other applications on your PC are concerned. 

[Bigsease30]

I think I understand. Let me give it a go and I will report back the results.

[srcrist]

Once that's done, you should be able to simply remove the drive letters from both your old pool and the CloudDrive. Neither of them need them. 

[srcrist]

It may actually be faster to move the content to the hidden poolpart folder in each individual drive's OLD poolpart folder. If that makes sense. 

[Bigsease30]

Followed the directions you provided in the previous post for enabling multi-format drives, rebooted and still can not see both the Gdrive and the Pool together. Here is the changes that I have made to the Settings.json file:

  "CoveFs_AllowMixedFilesystems": {
    "Default": false,
    "Override": true
  },

Without this option working, I am unable to move to the next step. Any other options?

[srcrist]

No. You'll have to get that working in order to accomplish this. Maybe submit a ticket to support. You probably won't hear back until Monday, though. 

[Bigsease30]

Ok, will do. Thanks for your help thus far.

 

[srcrist]

Sure thing. Good luck! Let me know how it turns out. I'm sure more people will want to try this over time. 

[Bigsease30]

Hey Srcrist, after several more reboots of the system and a little time, the Cloud Drive ended up showing as an available add after all. I have since decided to hold off on adding the Cloud drive until I complete the section below. 

On to my next issue. I disabled duplication and manually backups up my important files in-case of a mistake. I removed a single drive from the pool and reformatted it from ReFs to NTFS for compatibility and other personal reasons. When I add it back to the original pool, the stats on the left hand side still say ReFs but if I create a new pool with this drive, it shows as NTFS. I have since replicated this across 6 of my 14 drives so far and counting to get them all over to NTFS.

My question is this, If all of the drives are formatted and added back to the original pool, will the status update to NTFS if there are no ReFs drives left? I would rather not have to create a new pool and then move the data over and then have to link everything back to Z:\ manually. If it stays in the same pool, I will not have to do any extra work minus the drive removal, format and then re-add.

Thanks

[srcrist]

That's actually a really interesting question, and one I don't know the answer to. I have a pretty strong aversion to ReFS (had a large volume go RAW during a windows update once), so I haven't really used it enough to tell you. If I had to guess, yes. Once all of the drives are NTFS, I would imagine DrivePool would report the pool itself as NTFS. But, honestly, even if it *didn't*, you could always move the data out of poolpart folders on the drives and simply recreate the pool and move the data back--and then it would be NTFS for sure. So there isn't really a problem with continuing this path. That process would only take literal minutes, since Windows simply marks the data as moved without actually initiating a transfer. Moving data in and out of a poolpart folder on the same drive is a matter of seconds, generally speaking. 

One thing I want to make clear, though, is that we would still need a nested pool setup to accomplish the end goal. This is because the point of the nested pool isn't to provide some sort of compatibility with ReFS, it's to enable us to treat the entire pool of disks as a single volume to duplicate to another location. There simply isn't any way for us to add a CloudDrive volume to the same pool with all of the local drives, and tell DrivePool to place an additional copy of *all* of the data on the CloudDrive unless we treat the pool as a single volume to mirror. 

The good news is that hierarchical pooling is an in-built feature in DrivePool already. When you nest the pools, you'll see it make adjustments in the UI, signalling some operational adjustments that will ensure that performance isn't diminished. We aren't needlessly complicating the system, or making it do something it isn't *supposed* to do, this is just how the duplication logic of the pool needs to work in order to accomplish your goals. 

[Christopher (Drashna)]

On 2/25/2019 at 3:37 PM, Bigsease30 said:

My question is this, If all of the drives are formatted and added back to the original pool, will the status update to NTFS if there are no ReFs drives left? I would rather not have to create a new pool and then move the data over and then have to link everything back to Z:\ manually. If it stays in the same pool, I will not have to do any extra work minus the drive removal, format and then re-add.

 

Yes.

 

And by default, you can't mix NTFS and ReFS drives, unless you enable an advanced option.

 

Also, StableBit DrivePool should be smart enough to not read data from the cloud, if it's available locally.  The only exception is directory listings.

Quote

* [D] [Issue #27655] When a file is opened, each open context is given a priority. Read striping will always work over the highest 
                     available priority class. 
                     Priority classes from lowest to highest are:
                        - CloudDrive, CloudPool, LocalExternalDrive, NonCloudPool, StandardDrive

 

[Bigsease30]

Ok Friends,

The work on the hard drives is complete and data is back as it should be. The drivepool now identifies as NTFS.

I have created a second pool adding both the original pool as well as the google drive. I just cant figure out how to set the correct settings to allow the pool to duplicate to the Google Drive.

[srcrist]

BOTH the pool and the CloudDrive volume have to be allowed to accept *duplicated* data in the Drive Usage Limiter. If you set it so one can only hold duplicated data and one can only hold unduplicated data, the logic won't allow it to duplicate the content from one to the other. Make sense? So your pool should be able to hold BOTH, while your CloudDrive should only be able to accept duplicated data. Then it should replicate correctly. All NEW data will go to your Z: Pool, and the duplication settings will slowly copy that data to the CloudDrive over time. 

If duplicated data is disabled from a drive, it cannot store either the *original* copy NOR the second copy of a duplication. 

[Bigsease30]

3 minutes ago, srcrist said:

BOTH the pool and the CloudDrive volume have to be allowed to accept *duplicated* data in the Drive Usage Limiter. If you set it so one can only hold duplicated data and one can only hold unduplicated data, the logic won't allow it to duplicate the content from one to the other. Make sense? So your pool should be able to hold BOTH, while your CloudDrive should only be able to accept duplicated data. Then it should replicate correctly. All NEW data will go to your Z: Pool, and the duplication settings will slowly copy that data to the CloudDrive over time. 

If duplicated data is disabled from a drive, it cannot store either the *original* copy NOR the second copy of a duplication. 

Perfect! When I get home tonight I will implement the change and report back. Thanks again.

[srcrist]

Just now, Bigsease30 said:

Perfect! When I get home tonight I will implement the change and report back. Thanks again.

Sure thing. Two other things to note:

1) You'll also need to MOVE all of the existing data to the PoolPart folder created on Z: when you made the new pool. Otherwise it will not show up in the new pool for duplication to CloudDrive. 

2) The old pool no longer needs a drive letter. Move your Z: label to the NEW pool now, and applications will access the data via that. After you've moved the data to the new PoolPart folder, of course.