Jump to content
  • 0

Cloud Drive + G Suite = Backup disk


Bigsease30

Question

Hello All,

I have been using Drivepool for a long time now and love it. I am now looking to further protect my files so I purchased Cloud drive and G Suite Business today. I currently have 34.6TB of total space on my server with 8.71TB free. Duplication is currently turned on using (2X). I have connected my Google Drive to the cloud drive and created a 50TB drive. Assuming that this maybe to much now after figuring out that I am only actually using 13TB of space if I were to remove duplication from my current setup. I configured the download to use 10 threads and the upload to use 5 with an upload limit of 70mbps to avoid being banned.  To further add fuel to the fire, most of my data is movies and TV shows that currently are being utilized from Plex media player.

What I would like to achieve is to have a local copy of my files and have them dupe to my cloud drive. I would like for Plex to only use the local files and not attempt to re-download any files. Doing this would allow me to reclaim my duplicated space on my local server. I have tried to figure it out myself but have been struggling the last few hours. I don't want to accidentally delete my pool and lose everything. Any help would be appreciated. I have included screen shots of my drive pool and Cloud Drive. Please let me know if you need additional information to assist me.

Thanks in advance.

drivepool.PNG

clouddrive.PNG

Link to comment
Share on other sites

Recommended Posts

  • 0
2 hours ago, Bigsease30 said:

Hello again srcrist,

Thanks again for all the assistance that you have provided thus far. This community alone is one of the biggest reasons I keep referring Stablebit to friends and Colleges.

My performance settings are as follows.

Change your minimum download to your chunk size (probably 20MB, if you used the largest chunk possible). If you're just using this for backup, you really only need to engage the prefetcher when you're making large copies off the drive, so set it to 10MB in 10secs and have it grab maybe 100MB at a time. You can probably even disable it, if you want to. Keeping it enabled will basically only help smooth out network hiccups and help copies move smoother when you're copying data off of the drive. Other than that, you look good. 

Glad to help. Hope everything works out well for you. 

Link to comment
Share on other sites

  • 1
3 minutes ago, srcrist said:

BOTH the pool and the CloudDrive volume have to be allowed to accept *duplicated* data in the Drive Usage Limiter. If you set it so one can only hold duplicated data and one can only hold unduplicated data, the logic won't allow it to duplicate the content from one to the other. Make sense? So your pool should be able to hold BOTH, while your CloudDrive should only be able to accept duplicated data. Then it should replicate correctly. All NEW data will go to your Z: Pool, and the duplication settings will slowly copy that data to the CloudDrive over time. 

If duplicated data is disabled from a drive, it cannot store either the *original* copy NOR the second copy of a duplication. 

Perfect! When I get home tonight I will implement the change and report back. Thanks again.

Link to comment
Share on other sites

  • 1
Just now, Bigsease30 said:

Perfect! When I get home tonight I will implement the change and report back. Thanks again.

Sure thing. Two other things to note:

1) You'll also need to MOVE all of the existing data to the PoolPart folder created on Z: when you made the new pool. Otherwise it will not show up in the new pool for duplication to CloudDrive. 

2) The old pool no longer needs a drive letter. Move your Z: label to the NEW pool now, and applications will access the data via that. After you've moved the data to the new PoolPart folder, of course. 

Link to comment
Share on other sites

  • 1

That's OK. That just means you're out of space on your cache drive. It's normal when it's copying tons of data, and all it will do is slow down writes to the CloudDrive. You can only upload 750GB/day of data to Google anyway. (Which reminds me, throttle your upload to 70-75mbps or you'll get banned from the upload API for 24 hours.) The expandable cache will grow to use your entire drive for writes only. Read cache will still be limited to the amount that you set. If it's important for it not to take up the entire drive, you'll have to use a fixed cache--with the caveat that it will throttle writes to the drive as soon as the cache space fills up, instead of once the drive is out of room. So if you choose to use a fixed cache, give it a fairly generous cache. 

What are your CloudDrive performance settings? We might want to make some tweaks there to speed up the overall process. 

Link to comment
Share on other sites

  • 0
15 minutes ago, srcrist said:

OK, so this relates to the other thread. So if you ONLY want copies on the CloudDrive volume, you'll simply use the Drive Usage Limiter to enable only duplicated files on the CloudDrive volume, and then *disable* duplicated files on the local volumes. Then with 2x duplication, all of the second copies will be stored on the CloudDrive volume and only the CloudDrive volume. 

DrivePool is smart enough to use local storage for retrieval if it is available, over the CloudDrive volume. 

It will not mess up your pool. It will simply start migrating the data once a balancing pass begins. This process will, obviously, take a long time with that amount of data. Likely months. So be aware of that. 

Thanks for the reply. I found the location of the Drive usage limiter but how do I select the Cloud Drive as an option? It is not listed in the list of drives (see new screenshot). Under my drivepool, (see photo above) it doesn't list it to be able to be added to the original pool either.

DUL.PNG

Link to comment
Share on other sites

  • 0

I think what we actually need to do is create a NEW pool using your old pool and the CloudDrive volume. Disable duplication on your old pool entirely, and then enable 2x duplication in the new pool. That will duplicate the entire pool to the CloudDrive volume. Give that a test and see how it works. Though this doesn't solve the problem of your CloudDrive volume not showing up in DrivePool. I'm not sure why that would be. 

It might be because the pool is ReFS. Is the CloudDrive NTFS? I'm not sure if DrivePool can merge them. 

If you create a NEW pool, can you add a CloudDrive volume to that, in any case?

Link to comment
Share on other sites

  • 0

We don't want to remove them at all. We're creating a hierarchical pooling scheme. We are using your old pool to pool all of your local drives together, and then a new pool to duplicate the data between the pool and your CloudDrive. So leave your old pool as it is, but disable the 2x duplication on that pool. Then add your D: and your Z: to this new pool, and enable 2x duplication on that. 

Link to comment
Share on other sites

  • 0

Well the D: is not a pool. It's just a CloudDrive volume. You'll have two pools, which will be nested. 

Though, thinking about it, we'll need to also move all your existing data to the new poolpart folder that will be created on the old pool in order to make your existing data accessible to the NEW pool. So that may take a minute. 

Let me do this in a test with a smaller pool to make sure nothing crazy happens. Give me a minute. 

Link to comment
Share on other sites

  • 0

OK. So, here is the process as far as I can tell:

Create a new pool. Add your Z: and D: to the new pool. Disable duplication on your Z: pool entirely. Use the drive usage limiter on the NEW pool to set the CloudDrive volume to ONLY accept duplicated data. Move ALL existing content on Z: to the new poolpart folder on Z:, which shouldn't take long since DrivePool is smart enough to do it on the same drive. Once all of the data is accessible from the new pool, simply enable 2x duplication on the new pool, and it should slowly migrate all of the duplicates to D:.

Does that make sense?  

Once you're done with all of that, you can use Z: for the new pool and everything should be identical as far as the other applications on your PC are concerned. 

Link to comment
Share on other sites

  • 0

Followed the directions you provided in the previous post for enabling multi-format drives, rebooted and still can not see both the Gdrive and the Pool together. Here is the changes that I have made to the Settings.json file:

  "CoveFs_AllowMixedFilesystems": {
    "Default": false,
    "Override": true
  },

Without this option working, I am unable to move to the next step. Any other options?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...