Jump to content
Covecube Inc.
  • Announcements

    • Christopher (Drashna)

      Login issues   11/07/17

      If you have issues with logging in, make sure you use your display name and not the "username" or email.  Or head here for more info.   http://community.covecube.com/index.php?/topic/3252-login-issues/  
    • Christopher (Drashna)

      Getting Help   11/07/17

      If you're experiencing problems with the software, the best way to get ahold of us is to head to https://stablebit.com/Contact, especially if this is a licensing issue.    Issues submitted there are checked first, and handled more aggressively. So, especially if the problem is urgent, please head over there first. 
  • 0
KiLLeRRaT

Google Drive Rate Limits and Threading/Exp Backoff

Question

Hey Guys,

 

I got slow transfers with Google Drive, and set the threads to 12 up and 12 down.  This worked for a while and everything was a bit faster.

 

For the last two days, I have been getting countless Rate Limit Exceeded exceptions, even running 1 up and 1 down thread.

 

I check out online in the Google Drive API guides and found a bit about exponential backoff.

 

So a few questions/thoughts:

 

  • Is exponential backoff implemented for the Google Drive provider?
  • If I set the provider to use say 12 up and 12 down threads, do they all get blasted out using multiple requests at the same time? (causing rate limit exceptions later on)?
  • Would it work to have something like a 'request gatekeeper' where you can set your own rate limits client side so that no matter how many threads you run, it always obeys that limit you set, and so that there is a 'master exponential backoff' in place?

Is there at all a possibility to look at the provider implementation code? Or is this fully baked into the product? It'd be good if there was an API to allow anyone to build their own providers.

 

Thanks for a good product!

 

 

EDIT: Also, how will all the rate limiting work if you added say 5 Google Drives, each with 12 threads up and down? Quite quickly you will be making a TON of requests...

Share this post


Link to post
Share on other sites

39 answers to this question

Recommended Posts

  • 0

for those having issues with the 1.0.0.777 build,

 

What is you up and down stream? 

 

Also, have you tried changing the number of upload and download threads in the IO Performance? 

Also, have you tried reauthorizing the drive after upgrading to 1.0.0.777?

 

And if you've done all the above and are still having issues, could you do both of these? 

 

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

 

http://wiki.covecube.com/StableBit_CloudDrive_Web_Logs

Share this post


Link to post
Share on other sites
  • 0
Thank you for opening the logs! 

 

It looks like this may have been an issue with the Chunk ID system, and would have affected larger drives dispropotionately. 

 

Please grab the latest build, and see if that helps: 


Share this post


Link to post
Share on other sites
  • 0

Out of curiosity, is there any hope of Google increasing the call limit to a realistic value and/or any rough timeline of the "rewrite"?

 

The Amazon rate limit forced me to look into rclone for Amazon backup and now that I want to mirror some stuff from Google Drive to Amazon, I'm seeing how unstable the GD downloading is.  When I uploaded everything to GD, the speeds were fantastic (300-500 Mbps) but now when I try to download this same data from the same server, it generally hovers around 50 Mbps regardless of the version, settings, or server location.  I'm also getting throttling notifications almost constantly with only 10 download threads.

 

I'd love for Stablebit to be my permanent solution, and I've already purchased the full suite and a couple additional Cloud Drive licenses so I'm clearly in love with the products, I'm just hoping to get some encouraging news about the one thing making it unusable for me, the rate limiting (especially with GD which seems to have been fine until recently).

Share this post


Link to post
Share on other sites
  • 0

@Dejected,

 

Well, we're using a pool of API keys now, so the API limit should be much more reasonable (but just FYI, the limit is 10,000 per 100 seconds, or roughy 100 per second).  It's not a small limit, either. 

 

But since we're using a pool of keys now, this shouldn't be an issue. 

 

 

So if you're using the 1.0.0.770+ build, you should be using this pool.  However, please make sure you're on the latest beta (1.0.0.782-784),  as this should fix the remaining Google Drive performance issues. 

Share this post


Link to post
Share on other sites
  • 0

@Dejected,

 

Well, we're using a pool of API keys now, so the API limit should be much more reasonable (but just FYI, the limit is 10,000 per 100 seconds, or roughy 100 per second).  It's not a small limit, either. 

 

But since we're using a pool of keys now, this shouldn't be an issue. 

 

 

So if you're using the 1.0.0.770+ build, you should be using this pool.  However, please make sure you're on the latest beta (1.0.0.782-784),  as this should fix the remaining Google Drive performance issues. 

Thank you! I'd also like to report back on some things I've tried over the last few days, and I can gladly say that my issues were/are likely due to poor usage on my part.  I'm now at a spot where my download speed is between double and triple of what it was before and the rate limiting is virtually non existent with the download threads set at 20 (Prefetch forward = 20MB and Prefetch time = 120 seconds).

 

1. I was able to find various folders that ultimately had a bunch of smaller files (few thousand combined) which I've now downloaded and deleted from the drive.

2. I ran the cleanup option on the drive (Drive Options -> Cleanup).  Whenever you get a chance, would it be possible to provide insight on the role this plays and whether there are any recommendations on how often it should be used (probably dependent on drive size and usage?)?

 

My guess is that this won't be limited to GD so hopefully one or both of the things I did will also help others at some point.

Share this post


Link to post
Share on other sites
  • 0

I'm glad to hear it!

 

For the cleanup option, IIRC, this purges unused/unneed data on the provider, Especially in the case of shrinking a drive.  

 

Generally, you shouldn't need to use this option, but it may be a good idea to do so after resizing a drive. 

Share this post


Link to post
Share on other sites
  • 0

Anybody else having this issue on the .802 build? I'm in my trial period and I'm testing running a media server off of this before I buy it, I keep getting the "User Rate Limit Exceeded" error when I view the service log though. IT says it's uploading content, but I can't stream anything at all from it, like there's no down link.

Share this post


Link to post
Share on other sites
  • 0

I have a different problem, .802 build:

1:12:23.9: Warning: 0 : [ApiGoogleDrive:48] Google Drive returned error (quotaExceeded): The download quota for this file has been exceeded
1:12:24.0: Warning: 0 : [ApiHttp:48] HTTP protocol exception (Code=Forbidden).
1:12:24.0: Warning: 0 : [IoManager:48] HTTP error (Forbidden) performing I/O operation on provider.
1:12:24.0: Warning: 0 : [IoManager:48] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded

Not sure what to do with this, upload is stuck, logs don't show anything else since...

Share this post


Link to post
Share on other sites
  • 0

Try the latest build (1.0.0.846).  This should reset the trial period, as well. 

 

Once you've done that, reauthorize the drive ("Drive Options" -> "reauthorize").

This should help fix the issue with the user rate limit exceeded.

 

 

 

I have a different problem, .802 build:

1:12:23.9: Warning: 0 : [ApiGoogleDrive:48] Google Drive returned error (quotaExceeded): The download quota for this file has been exceeded
1:12:24.0: Warning: 0 : [ApiHttp:48] HTTP protocol exception (Code=Forbidden).
1:12:24.0: Warning: 0 : [IoManager:48] HTTP error (Forbidden) performing I/O operation on provider.
1:12:24.0: Warning: 0 : [IoManager:48] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded

Not sure what to do with this, upload is stuck, logs don't show anything else since...

 

This is an old drive, right? Eg, created on a much earlier version of StableBit CloudDrive? 

 

If so, try detaching and reattaching the drive. When reattaching the drive, click on "Advanced settings" and set the "Minimum download size" to 10MB, or higher. 

Share this post


Link to post
Share on other sites
  • 0

 

This is an old drive, right? Eg, created on a much earlier version of StableBit CloudDrive? 

 

 

Nope, I started testing CloudDrive on .802 so it was "brand new" drive. Before I saw the newer release I did try reauthorizing, detaching wasn't possible because of files waiting to upload (possibly "force detach" could help, or break it, didn't feel lucky there...).

But as the new update required a restart of the system anyway, it all started working again.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×