Jump to content
Covecube Inc.
  • 0
SirAce135

New convert! Need help planning :)

Question

Hello! 

I have just spent the last few hours researching and reading through this forum. I think I have decided to pull the trigger on CloudDrive I just need help planning.

I am currently using NetDrive to mount my GSuite google drive and it just seems to be choking.
I have a symmetrical gigabit connection and am running a pretty robust Plex Server along with a couple dozen 4K security cameras.

I currently have PLEX working with NetDrive and it works OK for the most part but the library scans take forever and 1 particular library just never completes. 

I currently have about 120TB of data already in my Google Drive so I would REALLY like to not have to re-upload it all.
I currently have the main GDrive mounted for Plex to use and metadata uploads (no real chance of hitting 750GB limit) and I have a Team Drive mounted for upload purposes so if it hits cap it doesn't affect Plex functionality.

So here come the questions:
1. Can I complete this migration without having to reupload everything?
2. I have read that pooldrive is used to overcome some windows 10 OS limitations in regards to large drives, my current mount is seen as a 2EB drive and they work "ok" am I missing something?
3. Through pooldrive, is it possible to pool the main Gdrive mount, and multiple Team drives to allow for round robin upload of files?
4. In a pooldrive with a GDrive and multiple Team drives, can you flag the GDrive as read only, and the team drives as upload only?
5. I have seen mention of detaching and reattaching drives for use. I Currently just mount the drive through NetDrive and start using it on all my computers and even mobile devices. Will I be able to still use it on the go like that?

Once I have a plan forward ironed out I will likely move on to confirm some setting recommendations I spotted elsewhere for ideal Plex use case settings. 

Thanks a ton in advance! I am really excited to see an active community and what seems like almost magical software to get these issues ironed out. If this all works out well I will likely buy a 10 pack of the license for all my computers. 

Share this post


Link to post
Share on other sites

6 answers to this question

Recommended Posts

  • 0

OK. So, there is a lot here, so let's unpack this one step at a time. I'm reading some fundamental confusion here, so I want to make sure to clear it up before you take any additional steps forward.

3 hours ago, SirAce135 said:

1. Can I complete this migration without having to reupload everything?

Starting here, which is very important: It's critical that you understand the distinction in methodology between something like Netdrive and CloudDrive, as a solution. Netdrive and rClone and their cousins are file-based solutions that effectively operate as frontends for Google's Drive API. They upload local files to Drive as files on Drive, and those files are then accessible from your Drive--whether online via the web interface, or via the tools themselves. That means that if you use Netdrive to upload a 100MB File1.bin, you'll have a 100MB file called File1.bin on your Google drive that is identical to the one you uploaded. Some solutions, like rClone, may upload the file with an obfuscated file name like Xf7f3g.bin, and even apply encryption to the file as it's being uploaded, and decryption when it is retrieved. But they are still uploading the entire file, as a file, using Google's API.

If you understand all of that, then understand that CloudDrive does not operate the same way.

CloudDrive is not a frontend for Google's API. CloudDrive creates a drive image, breaks that up into hundreds, thousands, or even millions of chunks, and then uses Google's infrastructure and API to upload those chunks to your cloud storage. This means that if you use CloudDrive to store a 100MB file called File1.bin, you'll actually have some number of chunks (depending on your configured chunk size) called something like XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-chunk-1, XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-chunk-2, XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX-chunk-3, etc, as well as a bunch of metadata that CloudDrive uses to access and modify the data on your drive. Note that these "files" do not correspond to the file size, type, or name that you uploaded, and cannot be accessed outside of CloudDrive in any meaningful way. CloudDrive actually stores your content as blocks, just like a physical hard drive, and then stores chunks of those blocks on your cloud storage. Though it can accomplish similar ends to Netdrive, rClone, or any related piece of software, its actual method of doing so is very different in important ways for you to understand.

So what, exactly, does this mean for you? It means, for starters, that you cannot simply use CloudDrive to access information that is already located in your cloud storage. CloudDrive only accesses information that has been converted to the format that it uses to store data, and CloudDrive's format cannot be accessed by other applications (or Google themselves). Any data that you'd like to migrate from your existing cloud storage to a CloudDrive volume must be downloaded and moved to the CloudDrive volume just as it would need to be if you were to migrate the data to a new physical drive on your machine--for the same reasons.

It may be helpful to think of CloudDrive as a virtual machine drive image. It's the same general concept. Just as you would have to copy data within the virtual machine in order to move data to the VM image, you'll have to copy data within CloudDrive to move it to your CloudDrive volume.

There are both benefits and drawbacks to using this approach:

Benefits  

  • CloudDrive is, in my experience, faster than rClone and its cousins. Particularly around the area of jumping to granular data locations, as you would for, say, jumping to a specific location in a media file.
  • CloudDrive stores an actual file system in the cloud, and that file system can be repaired and maintained just like one located on a physical drive. Tools like chkdsk and windows' own in-built indexing systems function on a CloudDrive volume just as they will on your local drive volumes. In your case this means that Plex's library scans will take seconds, and will not lock you out of Google's API limitations.
  • CloudDrive's block-based storage means that it can modify portions of files in-place, without downloading the entire file and reuploading it.
  • CloudDrive's cache is vastly more intelligent than those implemented by file-based solutions, and is capable of, for example, storing the most frequently accessed chunks of data, such as those containing the metadata information in media files, rather than whole media files. This, like the above, also translates to faster access times and searches.
  • CloudDrive's block-based solution allows for a level of encryption and data security that other solutions simply cannot match. Data is completely AES encrypted before it is ever even written to the cache, and not even Covecube themselves can access the data without your key. Neither your cloud provider, nor unauthorized users and administrators for your organization, can access your data without consent. 

Drawbacks (read carefully) 

  • CloudDrive's use of an actual file system also introduces vulnerabilities that file-based solutions do not have. If the file system data itself becomes corrupted on your storage, it can affect your ability to access the entire drive--in the same way that a corrupted file system can cause data loss on a physical drive as well. The most common sorts of corruption can be repaired with tools like chkdsk, but there have been incidents caused by Google's infrastructure that have caused massive data loss for CloudDrive users in the past--and there may be more in the future, though CloudDrive has implemented redundancies and checks to prevent them going forward. Note that tools like testdisk and recuva can be used on a CloudDrive volume just as they can on a physical volume in order to recover corrupt data, but this process is very tedious and generally only worth using for genuinely critical and irreplaceable data. I don't personally consider media files to be critical or irreplaceable, but each user must consider their own risk tolerance.
  • A CloudDrive volume is not accessible without CloudDrive. Your data will be locked into this ecosystem if you convert to CloudDrive as a solution. Your data will also only be accessible from one machine at a time. CloudDrive's caching system means that corruption can occur if multiple machines could access your data at once, and, as such, it will not permit the volume to be mounted by multiple instances simultaneously.
  • And, as mentioned, all data must be uploaded within the CloudDrive infrastructure to be used with CloudDrive. Your existing data will not work.

 

So, having said all of that, before I move on to helping you with your other questions, let me know that you're still interested in moving forward with this process. I can help you with the other questions, but I'm not sure that you were on the right page with the project you were signing up for here. rClone and NetDrive both also make fine solutions for media storage, but they're actually very different beasts than CloudDrive, and it's really important to understand the distinction. Many people are not interested in the additional limitations. 

Share this post


Link to post
Share on other sites
  • 0

Oh boy lol there's certainly a lot there and I absorbed and understood it all.

Hmmmm... It feels like there is a WHOLE lotta risk with not insignificant gain to be had with this system. 

Though I have to say that the combination of additional data loss risk combined with inability to access the data from multiple locations are probably my two largest hurdles to overcome. 

From a technical standpoint what you described is quite fascinating and I still want to poke around with. Perhaps not with my primary data hoard, and not with my company assets either. 

Can you still answer this question?
3. Through pooldrive, is it possible to pool the main Gdrive mount, and multiple Team drives to allow for round robin upload of files?

 

Thank you

Share this post


Link to post
Share on other sites
  • 0

CloudDrive itself does not support Team Drives because their API access is different. But DrivePool can certainly pool multiple CloudDrive volumes together. It can pool any volume your system can access.

But CloudDrive will not work to use Team Drives to evade the Google upload limitations, if that was your intention. There is some news that they are banning accounts for doing so, as well. Just FYI. See here: https://old.reddit.com/r/DataHoarder/comments/emuu9l/google_gsuit_whats_it_like_and_is_it_still_worth/fdw1cri/

I also want to add that the other solutions are not immune to the data loss issue any more than CloudDrive, it's just that data loss can manifest differently. If a file is corrupted on Google's end with rClone or Netdrive, you lose that file. If a chunk containing CloudDrive's file system is corrupted, you can lose access to the file system itself, and have to rebuild it. Ultimately, though, nobody should ever use cloud storage for any data that they consider to be irreplaceable without backups. 

Share this post


Link to post
Share on other sites
  • 0

Thanks for the info.
I meet and exceed the 5 user GSuite requirement and leverage those legitimate accounts to upload 750GB per day from each. I don't always hit that cap but it has happened. I Keep 1 account strictly for Plex playback and access purposes and the rest for round robin upload to load balance the uploads. 

Once again thanks a ton for all the work you did to clarify some misconceptions I had about CloudDrive. I can now see roughly how it works and why it works that way, I want to still play with it but I don't think it will be an ideal solution for my primary storage needs since I need to be able to access my data on the go from laptops and mobile devices.

Super cool idea and implementation though :)

Share this post


Link to post
Share on other sites
  • 0

Sure thing. Best of luck with finding something that works. If you're looking for a file based solution, it's honestly difficult to do better than rClone. I would look that direction. 

Share this post


Link to post
Share on other sites
  • 0

Thanks! 
Yeah I am playing with rClone now and am trying to dial in the mount cache options to provide the best responsiveness. 
I can provide it 1TB or so of SSD cache space but it doesn't seem to keep stuff around long enough to make it useful.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...