Jump to content
  • 0

Google Drive Rate Limits and Threading/Exp Backoff

KiLLeRRaT

Asked by KiLLeRRaT,

 Share

Question

KiLLeRRaT Member

KiLLeRRaT

Posted
Posted

Hey Guys,

 

I got slow transfers with Google Drive, and set the threads to 12 up and 12 down.  This worked for a while and everything was a bit faster.

 

For the last two days, I have been getting countless Rate Limit Exceeded exceptions, even running 1 up and 1 down thread.

 

I check out online in the Google Drive API guides and found a bit about exponential backoff.

 

So a few questions/thoughts:

 

  • Is exponential backoff implemented for the Google Drive provider?
  • If I set the provider to use say 12 up and 12 down threads, do they all get blasted out using multiple requests at the same time? (causing rate limit exceptions later on)?
  • Would it work to have something like a 'request gatekeeper' where you can set your own rate limits client side so that no matter how many threads you run, it always obeys that limit you set, and so that there is a 'master exponential backoff' in place?

Is there at all a possibility to look at the provider implementation code? Or is this fully baked into the product? It'd be good if there was an API to allow anyone to build their own providers.

 

Thanks for a good product!

 

 

EDIT: Also, how will all the rate limiting work if you added say 5 Google Drives, each with 12 threads up and down? Quite quickly you will be making a TON of requests...

Link to comment
Share on other sites

39 answers to this question

Recommended Posts

  • 0
Christopher (Drashna) Advanced Member

Christopher (Drashna)

Posted
Posted

Alex has released a new build that should address (or at least help minimize) this issue. 

 

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

 

Please download 1.0.0.725 1.0.0.749 if you're experiencing this issue. 

http://dl.covecube.com/CloudDriveWindows/beta/download/

Link to comment
Share on other sites
  • 0
Utnes Newbie

Utnes

Posted
Posted

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

Any chance to let advanced users use their own API credentials to bypass this?

Link to comment
Share on other sites
  • 0
Utnes Newbie

Utnes

Posted
Posted

Same here, seems that the error message is just hidden from users now.

 

Logs are flooded with:

1:58:22.6: Warning: 0 : [ApiGoogleDrive:57] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
1:58:22.6: Warning: 0 : [ApiHttp:57] HTTP protocol exception (Code=ServiceUnavailable).
1:58:22.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,903ms and retrying.
1:58:24.3: Warning: 0 : [IoManager:63] Error performing I/O operation on provider. Retrying. The request was aborted: The request was canceled.
Link to comment
Share on other sites
  • 0
Christopher (Drashna) Advanced Member

Christopher (Drashna)

Posted
Posted

Any chance to let advanced users use their own API credentials to bypass this?

 

It's possible. Though, it would depend on the authorization process. I'll ask Alex. 

 

725 seemed to just change the error I'm getting. Lots (in the thousands) of 'This thread was aborted. This thread was canceled.' 

 

 

 

Same here, seems that the error message is just hidden from users now.

 

Logs are flooded with:

1:58:22.6: Warning: 0 : [ApiGoogleDrive:57] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
1:58:22.6: Warning: 0 : [ApiHttp:57] HTTP protocol exception (Code=ServiceUnavailable).
1:58:22.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,903ms and retrying.
1:58:24.3: Warning: 0 : [IoManager:63] Error performing I/O operation on provider. Retrying. The request was aborted: The request was canceled.

 

 

 

This is because the fix is to change how the requests are handled. They're handled as throttling requests now, and the attempts are aborted and retried later. 

Link to comment
Share on other sites
  • 0
Christopher (Drashna) Advanced Member

Christopher (Drashna)

Posted
Posted

My understanding is that we are getting these errors because of the "app" is reaching it's api calls limit not the user? 

 

If so the more users CloudDrive has the "faster" we hit this limit?

 

Essentially, yeah.  However, since they're being handled as throttling requests, that means that the software starts backing off on requests (exponentially).  

 

The more people on this build means that the better it should be handled overall.

 

 

It also means that Alex is going to look into this very soon to see if we can optimize this or change things, so that we minimize API usage. 

Link to comment
Share on other sites
  • 0
nopnop Newbie

nopnop

Posted
Posted

hey there, after having lots of problems with throttled downloads I am now using the latest build .730 (updated from .725) since yesterday evening (GMT+1) but it got even worse with that version - google seems to be throttling the downloads practially *all the time* rendering my drives to an almost unusable state now. manually copying files with total commander take ages to complete, using them with plex is impossible already. the drives even get dismounted/remounted which was never the case before. it seems the increasing number of users (?) use up the application wide api limits imposed by google way too fast. please, for gods sake, try to fix that problem by any means - maybe give the users the opportunity to enter their own application keys to get their "own limits" instead of having to share the overall limits with all clouddrive users.

 

i really love clouddrive but it seems by now we are coming to a time where it really gets killed by its many users :(

 

(.725 doesnt really work anymore aswell so I am pretty much left with no access to the files by now)

12:40:56.3: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:40:56.3: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:40:56.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 4,613ms and retrying.
12:41:00.9: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:00.9: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:00.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 8,846ms and retrying.
12:41:09.9: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:09.9: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:09.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 16,406ms and retrying.
12:41:20.7: Warning: 0 : [ApiGoogleDrive:46] Google Drive returned error (internalError): Internal Error
12:41:20.7: Warning: 0 : [ApiHttp:46] HTTP protocol exception (Code=InternalServerError).
12:41:20.7: Warning: 0 : [IoManager:46] HTTP error (InternalServerError) performing I/O operation on provider.
12:41:20.7: Warning: 0 : [IoManager:46] Error performing I/O operation on provider. Retrying. Internal Error
12:41:20.8: Warning: 0 : [ApiGoogleDrive:22] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:20.8: Warning: 0 : [ApiHttp:22] HTTP protocol exception (Code=ServiceUnavailable).
12:41:20.8: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,179ms and retrying.
12:41:21.2: Warning: 0 : [CloudDrives] Cannot mount cloud part xxxxxxxx-cb81-4d0a-a1d7-xxxxxxxx. It was force unmounted.
12:41:26.3: Warning: 0 : [ApiGoogleDrive:74] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:26.3: Warning: 0 : [ApiHttp:74] HTTP protocol exception (Code=ServiceUnavailable).
12:41:26.3: Warning: 0 : [IoManager:74] HTTP error (ServiceUnavailable) performing I/O operation on provider.
12:41:26.3: Warning: 0 : [IoManager:74] Error performing I/O operation on provider. Retrying. Rate Limit Exceeded
12:41:26.4: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:26.4: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:26.4: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,039ms and retrying.
12:41:27.5: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:27.5: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:27.5: Warning: 0 : [ApiHttp] Server is throttling us, waiting 2,322ms and retrying.
12:41:29.9: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:29.9: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:29.9: Warning: 0 : [ApiHttp] Server is throttling us, waiting 4,791ms and retrying.
12:41:34.8: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:34.8: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:34.8: Warning: 0 : [ApiHttp] Server is throttling us, waiting 8,334ms and retrying.
12:41:43.3: Warning: 0 : [ApiGoogleDrive:61] Google Drive returned error (rateLimitExceeded): Rate Limit Exceeded
12:41:43.3: Warning: 0 : [ApiHttp:61] HTTP protocol exception (Code=ServiceUnavailable).
12:41:43.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 16,164ms and retrying.
12:41:51.5: Warning: 0 : [ApiGoogleDrive:22] Google Drive returned error (internalError): Internal Error
12:41:51.5: Warning: 0 : [ApiHttp:22] HTTP protocol exception (Code=InternalServerError).
12:41:51.5: Warning: 0 : [IoManager:22] HTTP error (InternalServerError) performing I/O operation on provider.
12:41:51.5: Warning: 0 : [IoManager:22] Error performing I/O operation on provider. Retrying. Internal Error
12:41:59.6: Warning: 0 : [IoManager:61] Thread abort performing I/O operation on provider.
12:41:59.6: Warning: 0 : [IoManager:61] Error processing read request. Der Thread wurde abgebrochen.
12:41:59.6: Warning: 0 : [IoManager:61] Error in read thread. Der Thread wurde abgebrochen.
Link to comment
Share on other sites
  • 0
drewleague Newbie

drewleague

Posted
Posted

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.

Link to comment
Share on other sites
  • 0
steffenmand Advanced Member

steffenmand

Posted
Posted

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.

Just disable the auto dismount thing in the advanced settings - that did it for me :-)

Link to comment
Share on other sites
  • 0
Christopher (Drashna) Advanced Member

Christopher (Drashna)

Posted
Posted

Are there any plans to actually address this issue? Your "fix" has done nothing, its just been swetp under the rug.

 

I'm a new user and would love to purchase a license, i thought i found what id been looking for all this time. I started using the software two weeks ago and now feel like the information ive managed to upload to my Google Drive is somewhat useless. As a previous user stated updating to version 730 has made it even worse so i downgraded back to 726 which seemed to be causing less issues.

 

Im getting constant dismounts and sometimes remounting will take 30 minutes or longer to finally connect. The software is borderline unusable at this point. Not to mention that im uploading this data at 1/7 of my available bandwidth, and its not about theoretical speeds or an mb to MB conversion, since i can get the speeds through other means.

 

This seems like a major issue, and it doesn't seem that theres been much of a response from your development team, at least not publicly.

 

Is there any workaround or improvement planned in the immediate near future? If we shouldn't continue to upload data via this avenue i think you should tell us that as users.

 

 

Yes, we do plan on fixing it. 

 

The 730 build is progress on this front, to optimize the cache so we can reduce the number of API calls that we make. 

 

Additionally, once this is finished, we plan on appealing/applying to Google for a rate increase, so we're not at the 10,000 requests per 100 seconds.  (breaks down to 100 requests per second). 

But we dont' want to submit until we have this change, so that Google can make their decision based on the much more optimized build. 

(eg submit the best build)

 

 

And this is a top priority for us, right now, as this fix can significantly help *every* provider, not just Google Drive (and is something that Alex was thinking about implementing anyways).

 

 

Just disable the auto dismount thing in the advanced settings - that did it for me :-)

 

http://wiki.covecube.com/StableBit_CloudDrive_Advanced_Settings

 

Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "0" and reboot to disable. 

 

Be warned, disabling this can cause the drive and system to lock up if there are too many issues. That's why the setting was added.

 

I'd actually recommend setting this to a higher value first and see how that works out, rather than disabling it outright. 

Link to comment
Share on other sites
  • 0
thatvalis Member

thatvalis

Posted
Posted

What is the latest recommended build and settings for best performance of Google Drive?

Tried 748, 730 and 725 yesterday but they all kept unmounting my drive.

Rolled back to 463 before going to bed and it has been going all night (although a bit slow).

 

Can you avoid the autodisounting if I change:

"Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "0" and reboot to disable."

?

 

Is there any risk of corrupting the data doing that?

Also, do you plan to add larger chunks in the future - and if so, will you be able to change the structure of your drive or do you need to start over?

Link to comment
Share on other sites
  • 0
thatvalis Member

thatvalis

Posted
Posted

Another idea that I had could be to allow for upload (and download) throttling per thread.

Currently the throttling in newer builds work for the total upload speed.

I.e if you set upload throttling to 20 mbps and have 4 threads open, then they are going to share the bandwidth.

My problem is that my uploads threads start and finish too quickly since I have 50 mbps upload (2 upload threads) - this leads to too many API calls and uploads stopping completely.

 

I was thinking if you could set for instance each thread can only use 1 mbps, then you can open several more threads going up at a slow pace.

That might lead to fewer API calls.

20 threads going at 1 mbps or lower with large chunks instead of 2 threads instantly completing and restarting at 50 mbps might be less stressful on the API.

Could be worth exploring maybe.

Link to comment
Share on other sites
  • 0
Christopher (Drashna) Advanced Member

Christopher (Drashna)

Posted
Posted

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.

 

Possible? Yes.  A good idea? That's the question.  How google will respond to that. 

 

The other issue is that we'd rather fix the issue for everyone, rather than issue a workaround.   Because even with a workaround, the issue could still occur (or google could crack down on API IDs).

 

Another idea that I had could be to allow for upload (and download) throttling per thread.

Currently the throttling in newer builds work for the total upload speed.

I.e if you set upload throttling to 20 mbps and have 4 threads open, then they are going to share the bandwidth.

My problem is that my uploads threads start and finish too quickly since I have 50 mbps upload (2 upload threads) - this leads to too many API calls and uploads stopping completely.

 

I was thinking if you could set for instance each thread can only use 1 mbps, then you can open several more threads going up at a slow pace.

That might lead to fewer API calls.

20 threads going at 1 mbps or lower with large chunks instead of 2 threads instantly completing and restarting at 50 mbps might be less stressful on the API.

Could be worth exploring maybe.

 

Actually, using larger partial chunk sizes will actually help with this.   It should allow for more data to be accessed sequentially. 

 

Changing the settings per thread wouldn't help, because the issues would still occur when the chunk is done. 

Link to comment
Share on other sites
  • 0
thnz Advanced Member

thnz

Posted
Posted

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.

 

Doing that might have unforeseen consequences. Google throttles by application for a reason, and bypassing that could potentially make their service worse for everyone. Amazon made their API invite only shortly after CloudDrive allowed custom profiles (not saying that the two were necessarily related), making it now much harder for other applications to integrate Amazon Drive support as a result.

Link to comment
Share on other sites
  • 0
wid_sbdp Advanced Member

wid_sbdp

Posted
Posted

Generally when these services provide developer API it's provided with the intent that it's utilized by the developer. Countless other apps utilize one API key and work with them to ensure stable operation of their application. That's kind of the whole point, as the registration to "develop" a new app on say ACD is pretty detailed and not something that is intended for every user of a particular service to be doing. So while it may work in the short term, and people may see better performance, there's probably a decent chance it would catch the eye of Amazon/Google/etc and they'd be like "woah hold on, why are there 900 CloudDrive projects going on right now?" That could lead to strict sanctions being placed on the applications ability to access the API.... because if there's an issue who does say Amazon contact now? You? Me? CloudBit? We're all registered as running that app...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
 Share
Go to question listing
×
×
  • Create New...