Jump to content
Covecube Inc.
  • 0

Limitation Settings?


Question

Ok, I realize this is a loaded question and potentially outside the scope of what CloudDrive was intended for.  I am curious as to the "official" answer though.  The online storage providers I use, Box, DropBox, Google, OneDrive, etc, are free accounts, not paid.  Most of them have specific limitations to how much can be uploaded in a day/week/month, maximum file size, maximum upload size, etc.  I would like to create drives to each of these providers and then pool them with DrivePool.  My concern is that DrivePool tries to copy/upload a file(s) to a provider that is beyond the limit.  Are there rules/plugins that I can configure for that pool/drive?

 

Here's a possible scenario:

I have a 50gb free Box account.  The limitation are a max file size of 250mb.

I have a 2gb free DropBox account.  200mb file size limitation and a maximum upload of 500mb per day.

I have a 200gb GoogleDrive account.  No llmitations.

 

- I want to use them in the above order, and only use the next drive if it doesn't meet the accounts criteria.  In that, if I have a 300mb file it would get put on the GoogleDrive account because it's too large for Box and DropBox.

- If I have a 20mb file, but Box is full, then it should go on DropBox.

- If I have a 5mb file and Box is not full, then the file should go on Box.

 

Can CloudDrive parse these type of "errors/warnings" from the provider to cease writes until defined configuration has been reached?

-if error "too big" then try next drive in pool.

-if error "too much uploaded" then start defined timer (ex. 24 days/hours/min) and try next drive.  This drive will cease to attempt writes until the defined time has lapsed.

 

Ultimately, I'm curious if this is something that can be configured in CloudDrive and passed to DrivePool, and/or controlled by rules in either.

Thanks!

Chris

Link to post
Share on other sites

3 answers to this question

Recommended Posts

  • 0

Well, the newer versions of StableBit CloudDrive include a bandwidth limit (upload and download) in the I/O Performance, which may help out with this. 

 

 

Additionally, when creating the CloudDrive disk, we use chunks. It's not a single, contiguous file, but potentially hundreds of small parts.  For instance, Google Drive we use up to 20MB chunks that are stored on the provider itself.  This is to prevent issues with file size limitations.   

 

And to clarify, the files that you're storing on the drive are not accessible on the provider. We store raw disk data (obfuscated via "easy encryption" to prevent issues with MIME types and content indexing on the provider side, as providers such as Google Drive and Amazon Cloud Drive do this). 

 

 

Additionally, that is part of why we have the cache. Data is stored in the cache first, and then uploaded. If it fails, we back off, and retry.  If the provider gives us a retry time, we honor it. Otherwise, we back off exponentially (meaning that we significantly increase the retry time to prevent hammering the provider).  

In the case with the DropBox provider, it should be fine handling things automatically, but you may notice upload errors in some circumstances. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...