Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Google Drive Rate Limits and Threading/Exp Backoff

ratelimit google drive api exponential backoff

Best Answer Christopher (Drashna) , 09 October 2016 - 06:41 PM

Alex has released a new build that should address (or at least help minimize) this issue. 

 

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

 

Please download 1.0.0.725 1.0.0.749 if you're experiencing this issue. 

http://dl.covecube.c.../beta/download/

Go to the full post


  • Please log in to reply
39 replies to this topic

#1 KiLLeRRaT

KiLLeRRaT

    Member

  • Members
  • PipPip
  • 23 posts

Posted 06 October 2016 - 12:29 AM

Hey Guys,

 

I got slow transfers with Google Drive, and set the threads to 12 up and 12 down.  This worked for a while and everything was a bit faster.

 

For the last two days, I have been getting countless Rate Limit Exceeded exceptions, even running 1 up and 1 down thread.

 

I check out online in the Google Drive API guides and found a bit about exponential backoff.

 

So a few questions/thoughts:

 

  • Is exponential backoff implemented for the Google Drive provider?
  • If I set the provider to use say 12 up and 12 down threads, do they all get blasted out using multiple requests at the same time? (causing rate limit exceptions later on)?
  • Would it work to have something like a 'request gatekeeper' where you can set your own rate limits client side so that no matter how many threads you run, it always obeys that limit you set, and so that there is a 'master exponential backoff' in place?

Is there at all a possibility to look at the provider implementation code? Or is this fully baked into the product? It'd be good if there was an API to allow anyone to build their own providers.

 

Thanks for a good product!

 

 

EDIT: Also, how will all the rate limiting work if you added say 5 Google Drives, each with 12 threads up and down? Quite quickly you will be making a TON of requests...


  • Ginoliggime, KiaraEvirm and Antoineki like this

#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 08 October 2016 - 01:20 AM

Known issue, and Alex should be looking into this today or tomorrow (or at least, very soon), as this appears to be affecting a lot of users. 

 

Also, this may be a server side setting, that we need to change. But it may not be. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 Choruptian

Choruptian

    Member

  • Members
  • PipPip
  • 27 posts

Posted 08 October 2016 - 10:49 AM

.



#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 09 October 2016 - 06:41 PM   Best Answer

Alex has released a new build that should address (or at least help minimize) this issue. 

 

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

 

Please download 1.0.0.725 1.0.0.749 if you're experiencing this issue. 

http://dl.covecube.c.../beta/download/


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 Utnes

Utnes

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 10 October 2016 - 04:18 AM

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

Any chance to let advanced users use their own API credentials to bypass this?



#6 smdotio

smdotio

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 11 October 2016 - 05:36 AM

725 seemed to just change the error I'm getting. Lots (in the thousands) of 'This thread was aborted. This thread was canceled.' 



#7 Utnes

Utnes

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 11 October 2016 - 06:02 AM

Same here, seems that the error message is just hidden from users now.

 

Logs are flooded with:

1:58:22.6: Warning: 0 : [ApiGoogleDrive:57] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
1:58:22.6: Warning: 0 : [ApiHttp:57] HTTP protocol exception (Code=ServiceUnavailable).
1:58:22.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,903ms and retrying.
1:58:24.3: Warning: 0 : [IoManager:63] Error performing I/O operation on provider. Retrying. The request was aborted: The request was canceled.


#8 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 11 October 2016 - 06:04 PM

Any chance to let advanced users use their own API credentials to bypass this?

 

It's possible. Though, it would depend on the authorization process. I'll ask Alex. 

 

725 seemed to just change the error I'm getting. Lots (in the thousands) of 'This thread was aborted. This thread was canceled.' 

 

 

 

Same here, seems that the error message is just hidden from users now.

 

Logs are flooded with:

1:58:22.6: Warning: 0 : [ApiGoogleDrive:57] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
1:58:22.6: Warning: 0 : [ApiHttp:57] HTTP protocol exception (Code=ServiceUnavailable).
1:58:22.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,903ms and retrying.
1:58:24.3: Warning: 0 : [IoManager:63] Error performing I/O operation on provider. Retrying. The request was aborted: The request was canceled.

 

 

 

This is because the fix is to change how the requests are handled. They're handled as throttling requests now, and the attempts are aborted and retried later. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#9 RulliDidd

RulliDidd

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 11 October 2016 - 07:40 PM

My understanding is that we are getting these errors because of the "app" is reaching it's api calls limit not the user? 

 

If so the more users CloudDrive has the "faster" we hit this limit?



#10 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 12 October 2016 - 02:05 AM

My understanding is that we are getting these errors because of the "app" is reaching it's api calls limit not the user? 

 

If so the more users CloudDrive has the "faster" we hit this limit?

 

Essentially, yeah.  However, since they're being handled as throttling requests, that means that the software starts backing off on requests (exponentially).  

 

The more people on this build means that the better it should be handled overall.

 

 

It also means that Alex is going to look into this very soon to see if we can optimize this or change things, so that we minimize API usage. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#11 nopnop

nopnop

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 22 October 2016 - 08:44 AM

hey there, after having lots of problems with throttled downloads I am now using the latest build .730 (updated from .725) since yesterday evening (GMT+1) but it got even worse with that version - google seems to be throttling the downloads practially *all the time* rendering my drives to an almost unusable state now. manually copying files with total commander take ages to complete, using them with plex is impossible already. the drives even get dismounted/remounted which was never the case before. it seems the increasing number of users (?) use up the application wide api limits imposed by google way too fast. please, for gods sake, try to fix that problem by any means - maybe give the users the opportunity to enter their own application keys to get their "own limits" instead of having to share the overall limits with all clouddrive users.

 

i really love clouddrive but it seems by now we are coming to a time where it really gets killed by its many users :(

 

(.725 doesnt really work anymore aswell so I am pretty much left with no access to the files by now)

12:40:56.3: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:40:56.3: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:40:56.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 4,613ms and retrying.
12:41:00.9: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:00.9: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:00.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 8,846ms and retrying.
12:41:09.9: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:09.9: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:09.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 16,406ms and retrying.
12:41:20.7: Warning: 0 : [ApiGoogleDrive:46] Google Drive returned error (internalError): Internal Error
12:41:20.7: Warning: 0 : [ApiHttp:46] HTTP protocol exception (Code=InternalServerError).
12:41:20.7: Warning: 0 : [IoManager:46] HTTP error (InternalServerError) performing I/O operation on provider.
12:41:20.7: Warning: 0 : [IoManager:46] Error performing I/O operation on provider. Retrying. Internal Error
12:41:20.8: Warning: 0 : [ApiGoogleDrive:22] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:20.8: Warning: 0 : [ApiHttp:22] HTTP protocol exception (Code=ServiceUnavailable).
12:41:20.8: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,179ms and retrying.
12:41:21.2: Warning: 0 : [CloudDrives] Cannot mount cloud part xxxxxxxx-cb81-4d0a-a1d7-xxxxxxxx. It was force unmounted.
12:41:26.3: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:26.3: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:26.3: Warning: 0 : [IoManager:74] HTTP error (ServiceUnavailable) performing I/O operation on provider.
12:41:26.3: Warning: 0 : [IoManager:74] Error performing I/O operation on provider. Retrying. Rate Limit Exceeded
12:41:26.4: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:26.4: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:26.4: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,039ms and retrying.
12:41:27.5: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:27.5: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:27.5: Warning: 0 : [ApiHttp] Server is throttling us, waiting 2,322ms and retrying.
12:41:29.9: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:29.9: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:29.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 4,791ms and retrying.
12:41:34.8: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:34.8: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:34.8: Warning: 0 : [ApiHttp] Server is throttling us, waiting 8,334ms and retrying.
12:41:43.3: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:43.3: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:43.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 16,164ms and retrying.
12:41:51.5: Warning: 0 : [ApiGoogleDrive:22] Google Drive returned error (internalError): Internal Error
12:41:51.5: Warning: 0 : [ApiHttp:22] HTTP protocol exception (Code=InternalServerError).
12:41:51.5: Warning: 0 : [IoManager:22] HTTP error (InternalServerError) performing I/O operation on provider.
12:41:51.5: Warning: 0 : [IoManager:22] Error performing I/O operation on provider. Retrying. Internal Error
12:41:59.6: Warning: 0 : [IoManager:61] Thread abort performing I/O operation on provider.
12:41:59.6: Warning: 0 : [IoManager:61] Error processing read request. Der Thread wurde abgebrochen.
12:41:59.6: Warning: 0 : [IoManager:61] Error in read thread. Der Thread wurde abgebrochen.


#12 thnz

thnz

    Advanced Member

  • Members
  • PipPipPip
  • 139 posts

Posted 23 October 2016 - 09:17 AM

Isn't there a per-user requests/100s quota you can set? Perhaps you could turn it down in the meantime to keep things functional for as many users as possible - until you can increase the apps total daily limit.



#13 drewleague

drewleague

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 October 2016 - 01:26 PM

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.



#14 steffenmand

steffenmand

    Advanced Member

  • Members
  • PipPipPip
  • 296 posts

Posted 23 October 2016 - 08:41 PM

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.

Just disable the auto dismount thing in the advanced settings - that did it for me :-)



#15 drewleague

drewleague

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 October 2016 - 10:47 AM

Just disable the auto dismount thing in the advanced settings - that did it for me :-)

Is this only on a certain version because i cant seem to find it anywhere, still getting disconnects, now on version 726 its more often as well.



#16 steffenmand

steffenmand

    Advanced Member

  • Members
  • PipPipPip
  • 296 posts

Posted 24 October 2016 - 03:56 PM

Is this only on a certain version because i cant seem to find it anywhere, still getting disconnects, now on version 726 its more often as well.

You have to manually edit the config file - you can't do it through the UI.



#17 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 24 October 2016 - 09:02 PM

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.

 

 

Yes, we do plan on fixing it. 

 

The 730 build is progress on this front, to optimize the cache so we can reduce the number of API calls that we make. 

 

Additionally, once this is finished, we plan on appealing/applying to Google for a rate increase, so we're not at the 10,000 requests per 100 seconds.  (breaks down to 100 requests per second). 

But we dont' want to submit until we have this change, so that Google can make their decision based on the much more optimized build. 

(eg submit the best build)

 

 

And this is a top priority for us, right now, as this fix can significantly help *every* provider, not just Google Drive (and is something that Alex was thinking about implementing anyways).

 

 

Just disable the auto dismount thing in the advanced settings - that did it for me :-)

 

http://wiki.covecube...vanced_Settings

 

Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "0" and reboot to disable. 

 

Be warned, disabling this can cause the drive and system to lock up if there are too many issues. That's why the setting was added.

 

I'd actually recommend setting this to a higher value first and see how that works out, rather than disabling it outright. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#18 thatvalis

thatvalis

    Member

  • Members
  • PipPip
  • 10 posts

Posted 29 October 2016 - 06:15 AM

What is the latest recommended build and settings for best performance of Google Drive?

Tried 748, 730 and 725 yesterday but they all kept unmounting my drive.

Rolled back to 463 before going to bed and it has been going all night (although a bit slow).

 

Can you avoid the autodisounting if I change:

"Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "0" and reboot to disable."

?

 

Is there any risk of corrupting the data doing that?

Also, do you plan to add larger chunks in the future - and if so, will you be able to change the structure of your drive or do you need to start over?



#19 pocomo

pocomo

    Member

  • Members
  • PipPip
  • 17 posts

Posted 29 October 2016 - 03:52 PM

I am running build 730 on Win10 X64, with 10MB chunks and 3 threads in each direction. Not a high performance configuration but I haven't had a dismount in a week (with pretty constant usage).



#20 rusty0078

rusty0078

    Newbie

  • Members
  • Pip
  • 1 posts
  • LocationMissouri, USA

Posted 31 October 2016 - 03:40 AM

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.


  • thatvalis likes this





Also tagged with one or more of these keywords: ratelimit, google drive, api, exponential backoff

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users