Jump to content
Covecube Inc.
  • 0

Questions About Functionality / Use Case


Question

Hi!

I had a question about how CloudDrive works. I understand that there is a local cache stored on one of the drives, and I'm pretty sure that local cache is used for a temporary place while files are being uploaded and a local cache for frequently used files. I'm currently using CloudDrive with Amazon Cloud Drive, basically as a test as I'm curious about the data loss. I've had a fair amount of uploads fail from ACD_CLI, but retrying them always works.

My question is that since most of us probably have a large amount of drives on our severs I'm curious about the cache. I've seen the threads here stating that there is potential data loss with Amazon Cloud Drive. I'm curious that if there is a large enough cache (let's say 500gb, or even expandable in the later BETAs) how CloudDrive would handle the failures. My assumption is that if the upload fails to ACD CloudDrive would attempt to re-upload the file and try again. Is this possible? This would help avoid failures, and then once the file is in the cloud the local file is deleted. Perhaps it does this and I'm overlooking how it works.

I can see two potential problems with this:

  • The cache is used for frequent files as well (from my understanding) meaning that there would be ~500GB of local data. I'd personally want to avoid that. The idea I had was to keep the file locally ONLY until it's fully uploaded and verified in the cloud.
  • The way CloudDrive uploads files wouldn't allow for this due to how it uploads (it looks like it uploads in parts), but I'm not familiar with the file store.

My purpose of using CloudDrive is an easy way to have the cloud data mounted as a drive in Windows that is also encrypted and can easily mounted to other machines within the network.

Thanks!

Link to post
Share on other sites

1 answer to this question

Recommended Posts

  • 0

The Nuts and Bolts section has a lot of good info:

http://community.covecube.com/index.php?/forum/19-nuts-bolts/

 

Including a long write up by Alex (the Developer) about how the cache works:

http://community.covecube.com/index.php?/topic/1610-how-the-stablebit-clouddrive-cache-works/

 

 

 

Though, specifically, we create a virtual drive, we store the chunks of this drive on the cloud provider, and use the cache to keep some content local.  "To Upload" content is always cached until needed, and then released unless it's still needed locally, as dictated by disk access). 

 

As for Amazon Cloud Drive, we have an "upload verification" option that is enabled by default on this provider. This will redownload the newly uploaded file to ensure that it ACTUALLY uploaded.  If it succeeds, we release the chunk and move onto the next. if it fails, we reupload it and check again. 

 

 

As for "large enough cache", this depends.  The default cache type will expand past the specified size if you're uploading a lot. We've added a couple of additional cache types in the internal betas, because we've had more than a few people complain about this behavior. So the internal betas have an option that has a hard limit on the cache size, as well as other options. 

 

The cache is used for requested files (such as data grabbed by the prefetcher, which is a performance optimizer routine), and we try to keep commonly used data in the cache. For instance, we pin NTFS metadata by default. And we have an option to pin directory data (to help speed up directory enumeration). 

 

 

If you want files only to be cached locally until they're uploaded, you can set the cache to 0 bytes.  This will aggressively clear the cache.  However, I'd recommend 50-100GBs for the cache size. This should be plenty for NTFS metadata, as well as a good amount of "space" for prefetched data. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...