Jump to content
  • 0

Using in multiple systems


Datahoarder

Question

Greetings, I'm looking for your opinion regarding CloudDrive and how it matches what I'm looking for. I'll try to keep it brief.

 

I'm currently testing CloudDrive with Google Drive provider, which is what I would eventually use if I decide to go for it.

 

I want an encrypted cloud drive that would act as a normal drive on my system. So far it's a match.

 

I would also like to use said cloud drive on multiple systems, though not simultaneously. The idea is to use it on a home computer when I'm home and on a work computer when I'm not.

 

Now I know CloudDrive has the detach option, but every reattaching seems to take time and download loads of data. I assume that data is some sort of metadata and similar. After it shows up in system, it also starts downloading and "pinning data", which I guess is more detailed filesystem info. What I don't understand is it's size - already at 500 MB with 11.7 GB of data on the drive. What size would it be if I had 1 TB, 10 TB, 100 TB? How long would it actually take to attach and start to use it then? Is it even built for such usecase? If it isn't, what can I do apart from lowering my expectations?

 

Thank you for any opinions!

Link to comment
Share on other sites

8 answers to this question

Recommended Posts

  • 0

You have the ability to completely disable pinning of any data, you can individually "Pin directories" which will make listing of all files/folders be instantaneous and also "Pin metadata" which I'm unsure about but I'm guessing is file metadata such as time created to embedded album artwork. Because of this the accumulated data from pinning is determined by the amount of files/folders and the metadata size.

 

The detaching process is equally "slow" on all drives (AFAIK). Takes like 30 seconds? (Depends on bandwidth, API throttling, download/upload threads). Thus you should be able to just click detach, let it do its thing and move on to the other computer.

 

You can use the drive the second you see it available in a File Explorer. (On-par with detach time)

Link to comment
Share on other sites

  • 0

I'm assuming you mean that it's already downloaded 500MB of data when there is 11.7GB of data on the drive. 

 

Specifically, this is likely due to the file system data.

By default, all drives are configured to automatically pin NTFS file system structures. This is to significantly improve performance when reading and writing to the disk.  This data include stuff like the partition header (MBR vs GPT), the partition info (size, geometry, format type - NTFS vs ReFS vs exFAT vs etc), USN journal, partition table, the NTFS index, etc.  

 

Basically, all of the stuff in the "Physical structure" of the disk: 

https://technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx#w2k3tr_ntfs_how_dhao

 

This is not an insignificant amount of data, to be honest. 

 

One of the option settings is to pin the directory structure.  If this is enabled (which is a good idea), this then stores all of the directory entries on the volume.  Each of these is usually 4kb in size (or whatever the cluster or "allocation unit" size is). This are small, but can add up when you have hundreds, thousands or more folders on the drive. 

 

 

Besides these, Windows automatically creates a recycle bin on the drive, as well as the "System Volume Information" folder, which is used for VSS (previous versions, snapshots, backup, etc).  This is automatically accessed and generated, which may cause the contents to be downloaded immediately when mounting the drive. 

 

 

 

As for the attach process itself, it has to download the drive info first, it also downloads the Chunk ID database on newer versions. This is a full database, and can be large, depending on the size of the drive.

 

After this information is downloaded, then we can "mount" the disk in VDS (Virtual Disk Service).  This process can take a bit, as VDS isn't known for it's speed.  This is why Windows can take a minute or three when plugging in a disk before it pops up in the system. Our software is going through the exact same process. 

 

Once that happens, Windows will auto-mount the drive to a disk letter, which ... also relies on VDS. 

 

So, there are a lot of things we have to do before the drive can be completely mounted on the system.   We do try to streamline this process so it happens quickly, but that doesn't always happen.  

And if you're getting throttled when this happens, it will slow down the process, as well. 

 

This same "delay" can be seen with StableBit DrivePool, as well. Which is an emulated file system and doesn't have a lot of these issues (but also uses VDS). 

 

As for detaching the drive, this is always going to be a slow process.  The remaining "to upload" data has to be finished uploading before the drive can be unmounted.  We also have to close all of the handles to the drive ("open files", basically), and this can take a long time in some cases. and then the drive is unmounted in VDS and detached.  

Link to comment
Share on other sites

  • 0

 

 

I'm assuming you mean that it's already downloaded 500MB of data when there is 11.7GB of data on the drive. 

 

No. After attaching and showing up in system, I looked at the drive root and it automatically started "pinning data" on it's own. That ended up being 500MB of pinned data. This is why I was worried this huge block of data is necessarry to download after each attaching and that it could be the size of tens of gigabytes on a larger drive. Anyway, after unchecking Pin metadata and leaving Pin directories, it still does that but fortunately it's a lot less data.

 

Overall I find the whole "Pinning data" procedure somewhat mysterious and I wish it was explaining a bit more why it downloads what it downloads. I had the PC on overnight, the cloud drive was not touched by anything ( I don't even have windows search indexing service on), yet the first thing I saw in the morning was "Pinning data" - why when nothing was done to the drive?

 

Thanks for more specific explanation for attaching. So can I assume that attaching let's say full 10TB drive won't take something like 10-20 minutes, but some time not much longer than current drive with 10GB?

Link to comment
Share on other sites

  • 0

Ah, okay.

 

And yeah, the metadata is all of the volume and NTFS information for the disk.  This will vary with drive, but it shouldn't be significantly larger for larger drives.  Though, with larger drives (well, more used drives), this will definitely increase, but it shouldn't significantly increase. with the larger drives. 

 

 

How large is the drive, though? Not the used size, but the total capacity? 

 

 

And yeah, it definitely shouldn't take very long to attach a 10TB drive or larger.  Even with it being well used. It shouldn't take longer than a minute or two. 

Link to comment
Share on other sites

  • 0

This is why I was worried this huge block of data is necessarry to download after each attaching and that it could be the size of tens of gigabytes on a larger drive.

 

From my experience, it doesn't have as much to do with overall drive size, as much as it has to do with the number of files and directories on it. I have a large drive (200TB) and it has a little over 1000 files and about 100 folders adding up to about 350GB in data. That drive only needs about 570MB of space for pinned data (directories and metadata).

 

I also have a 10TB drive that has about 44 thousand files on it, over a thousand folders, all of that adding up to about 280GB, and it needs 700MB of space for pinned data. I'm guessing a large part of that size difference is index files, especially since a couple of those folders hold tens of thousands of files/folders.

 

And I can echo what Christopher said that it only takes a couple minutes to attach a large drive. I regularly switch those drives between 2 computers, and the first time I attached them to another computer only took a couple minutes before they finished mounting.

Link to comment
Share on other sites

  • 0

Just testing, it looks like at least Google Drive disks do automatically pin both metadata and the directory structure.  

 

If you have a lot of folders on the system, then yes, this would definitely increase the size used, as it's downloading a large chunk of data, in this case. 

 

 

@17thspartan:  Yes, the NTFS index is part of the mentioned metadata, and this is pinned as part of said metadata. 

Link to comment
Share on other sites

  • 0

The drive in question was put at 10TB (no particular reason, just wanted to try what happens with that size :) ). I've been testing it a bit more, so far it was actually pretty good. I wonder though if the default automatic pinning of metadata and directory structure is there for any specific reason related to Google Drive.... Because right now I have reached this problem: http://community.covecube.com/index.php?/topic/2228-google-drive-rate-limits-and-threadingexp-backoff/?p=18212

Link to comment
Share on other sites

  • 0

The pinning of metadata and directory structure is to specifically reduce API load and increase performance.

 

The NTFS metadata is accessed incredibly frequently, so keeping it in the cache means that it's always available. 

As for the directory structure, this is specifically to help speed up directory enumeration.

 

 

As for google drive rate limits, make sure you're on the latest public beta (1.0.0.846), and try reauthorizing the Google Drive. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...