Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Google Drive Rate Limits and Threading/Exp Backoff

ratelimit google drive api exponential backoff

Best Answer Christopher (Drashna) , 09 October 2016 - 06:41 PM

Alex has released a new build that should address (or at least help minimize) this issue. 

 

Specifically, it looks like this is an app wide limit on the number of API calls that needs to be "appealed" to Google to get increased, or a rewrite is needed to minimize the API calls. 

 

 

Please download 1.0.0.725 1.0.0.749 if you're experiencing this issue. 

http://dl.covecube.c.../beta/download/

Go to the full post


  • Please log in to reply
39 replies to this topic

#21 thatvalis

thatvalis

    Member

  • Members
  • PipPip
  • 10 posts

Posted 31 October 2016 - 07:56 AM

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.

 

That sounds like a great idea if it's possible!


  • Ginoliggime, KiaraEvirm and Antoineki like this

#22 thatvalis

thatvalis

    Member

  • Members
  • PipPip
  • 10 posts

Posted 31 October 2016 - 03:36 PM

Another idea that I had could be to allow for upload (and download) throttling per thread.

Currently the throttling in newer builds work for the total upload speed.

I.e if you set upload throttling to 20 mbps and have 4 threads open, then they are going to share the bandwidth.

My problem is that my uploads threads start and finish too quickly since I have 50 mbps upload (2 upload threads) - this leads to too many API calls and uploads stopping completely.

 

I was thinking if you could set for instance each thread can only use 1 mbps, then you can open several more threads going up at a slow pace.

That might lead to fewer API calls.

20 threads going at 1 mbps or lower with large chunks instead of 2 threads instantly completing and restarting at 50 mbps might be less stressful on the API.

Could be worth exploring maybe.



#23 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 31 October 2016 - 06:55 PM

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.

 

Possible? Yes.  A good idea? That's the question.  How google will respond to that. 

 

The other issue is that we'd rather fix the issue for everyone, rather than issue a workaround.   Because even with a workaround, the issue could still occur (or google could crack down on API IDs).

 

Another idea that I had could be to allow for upload (and download) throttling per thread.

Currently the throttling in newer builds work for the total upload speed.

I.e if you set upload throttling to 20 mbps and have 4 threads open, then they are going to share the bandwidth.

My problem is that my uploads threads start and finish too quickly since I have 50 mbps upload (2 upload threads) - this leads to too many API calls and uploads stopping completely.

 

I was thinking if you could set for instance each thread can only use 1 mbps, then you can open several more threads going up at a slow pace.

That might lead to fewer API calls.

20 threads going at 1 mbps or lower with large chunks instead of 2 threads instantly completing and restarting at 50 mbps might be less stressful on the API.

Could be worth exploring maybe.

 

Actually, using larger partial chunk sizes will actually help with this.   It should allow for more data to be accessed sequentially. 

 

Changing the settings per thread wouldn't help, because the issues would still occur when the chunk is done. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#24 thnz

thnz

    Advanced Member

  • Members
  • PipPipPip
  • 139 posts

Posted 31 October 2016 - 08:20 PM

Would it be possible to allow the use of your own Drive API client ids in CloudDrive?  If users were able to use their own ids then there would be no need to request a rate increase.

 

Doing that might have unforeseen consequences. Google throttles by application for a reason, and bypassing that could potentially make their service worse for everyone. Amazon made their API invite only shortly after CloudDrive allowed custom profiles (not saying that the two were necessarily related), making it now much harder for other applications to integrate Amazon Drive support as a result.



#25 wid_sbdp

wid_sbdp

    Advanced Member

  • Members
  • PipPipPip
  • 49 posts

Posted 01 November 2016 - 12:41 AM

Generally when these services provide developer API it's provided with the intent that it's utilized by the developer. Countless other apps utilize one API key and work with them to ensure stable operation of their application. That's kind of the whole point, as the registration to "develop" a new app on say ACD is pretty detailed and not something that is intended for every user of a particular service to be doing. So while it may work in the short term, and people may see better performance, there's probably a decent chance it would catch the eye of Amazon/Google/etc and they'd be like "woah hold on, why are there 900 CloudDrive projects going on right now?" That could lead to strict sanctions being placed on the applications ability to access the API.... because if there's an issue who does say Amazon contact now? You? Me? CloudBit? We're all registered as running that app...



#26 coolspot

coolspot

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 11 December 2016 - 07:02 AM

I assume this is still a problem? I'm using .777  and am having trouble uploading more than ~10 files in a single batch?



#27 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 11 December 2016 - 09:27 PM

for those having issues with the 1.0.0.777 build,

 

What is you up and down stream? 

 

Also, have you tried changing the number of upload and download threads in the IO Performance? 

Also, have you tried reauthorizing the drive after upgrading to 1.0.0.777?

 

And if you've done all the above and are still having issues, could you do both of these? 

 

http://wiki.covecube...e_Drive_Tracing

 

http://wiki.covecube...dDrive_Web_Logs


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#28 KiLLeRRaT

KiLLeRRaT

    Member

  • Members
  • PipPip
  • 23 posts

Posted 12 December 2016 - 07:37 PM

Hi Chris, I have just uploaded the drive trace.  I'm having an issue where my download is BARELY getting 3mbps using 8 download threads :(

 

I have a 130mbit connection.

 

Let me know if that helps.



#29 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 14 December 2016 - 07:09 PM

Thank you for opening the logs! 
 
It looks like this may have been an issue with the Chunk ID system, and would have affected larger drives dispropotionately. 
 
Please grab the latest build, and see if that helps: 

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#30 KiLLeRRaT

KiLLeRRaT

    Member

  • Members
  • PipPip
  • 23 posts

Posted 14 December 2016 - 07:17 PM

Excellent, thanks, I will give this a shot and let you know how it goes!



#31 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 14 December 2016 - 07:18 PM

Excellent, thanks, I will give this a shot and let you know how it goes!

 

Please do.  

 

If it doesn't help, then please grab the logs again, and let us know. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#32 dejected

dejected

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 15 December 2016 - 03:59 AM

Out of curiosity, is there any hope of Google increasing the call limit to a realistic value and/or any rough timeline of the "rewrite"?

 

The Amazon rate limit forced me to look into rclone for Amazon backup and now that I want to mirror some stuff from Google Drive to Amazon, I'm seeing how unstable the GD downloading is.  When I uploaded everything to GD, the speeds were fantastic (300-500 Mbps) but now when I try to download this same data from the same server, it generally hovers around 50 Mbps regardless of the version, settings, or server location.  I'm also getting throttling notifications almost constantly with only 10 download threads.

 

I'd love for Stablebit to be my permanent solution, and I've already purchased the full suite and a couple additional Cloud Drive licenses so I'm clearly in love with the products, I'm just hoping to get some encouraging news about the one thing making it unusable for me, the rate limiting (especially with GD which seems to have been fine until recently).



#33 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 16 December 2016 - 09:44 PM

@Dejected,

 

Well, we're using a pool of API keys now, so the API limit should be much more reasonable (but just FYI, the limit is 10,000 per 100 seconds, or roughy 100 per second).  It's not a small limit, either. 

 

But since we're using a pool of keys now, this shouldn't be an issue. 

 

 

So if you're using the 1.0.0.770+ build, you should be using this pool.  However, please make sure you're on the latest beta (1.0.0.782-784),  as this should fix the remaining Google Drive performance issues. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#34 dejected

dejected

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 17 December 2016 - 02:37 PM

@Dejected,

 

Well, we're using a pool of API keys now, so the API limit should be much more reasonable (but just FYI, the limit is 10,000 per 100 seconds, or roughy 100 per second).  It's not a small limit, either. 

 

But since we're using a pool of keys now, this shouldn't be an issue. 

 

 

So if you're using the 1.0.0.770+ build, you should be using this pool.  However, please make sure you're on the latest beta (1.0.0.782-784),  as this should fix the remaining Google Drive performance issues. 

Thank you! I'd also like to report back on some things I've tried over the last few days, and I can gladly say that my issues were/are likely due to poor usage on my part.  I'm now at a spot where my download speed is between double and triple of what it was before and the rate limiting is virtually non existent with the download threads set at 20 (Prefetch forward = 20MB and Prefetch time = 120 seconds).

 

1. I was able to find various folders that ultimately had a bunch of smaller files (few thousand combined) which I've now downloaded and deleted from the drive.

2. I ran the cleanup option on the drive (Drive Options -> Cleanup).  Whenever you get a chance, would it be possible to provide insight on the role this plays and whether there are any recommendations on how often it should be used (probably dependent on drive size and usage?)?

 

My guess is that this won't be limited to GD so hopefully one or both of the things I did will also help others at some point.


  • Christopher (Drashna) likes this

#35 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 01 January 2017 - 11:48 PM

I'm glad to hear it!

 

For the cleanup option, IIRC, this purges unused/unneed data on the provider, Especially in the case of shrinking a drive.  

 

Generally, you shouldn't need to use this option, but it may be a good idea to do so after resizing a drive. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#36 dmwilson220

dmwilson220

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 16 January 2017 - 03:47 PM

Anybody else having this issue on the .802 build? I'm in my trial period and I'm testing running a media server off of this before I buy it, I keep getting the "User Rate Limit Exceeded" error when I view the service log though. IT says it's uploading content, but I can't stream anything at all from it, like there's no down link.



#37 Datahoarder

Datahoarder

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 17 February 2017 - 03:43 PM

I have a different problem, .802 build:

1:12:23.9: Warning: 0 : [ApiGoogleDrive:48] Google Drive returned error (quotaExceeded): The download quota for this file has been exceeded
1:12:24.0: Warning: 0 : [ApiHttp:48] HTTP protocol exception (Code=Forbidden).
1:12:24.0: Warning: 0 : [IoManager:48] HTTP error (Forbidden) performing I/O operation on provider.
1:12:24.0: Warning: 0 : [IoManager:48] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded

Not sure what to do with this, upload is stuck, logs don't show anything else since...



#38 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 19 February 2017 - 03:04 AM

Try the latest build (1.0.0.846).  This should reset the trial period, as well. 

 

Once you've done that, reauthorize the drive ("Drive Options" -> "reauthorize").

This should help fix the issue with the user rate limit exceeded.

 

 

 

I have a different problem, .802 build:

1:12:23.9: Warning: 0 : [ApiGoogleDrive:48] Google Drive returned error (quotaExceeded): The download quota for this file has been exceeded
1:12:24.0: Warning: 0 : [ApiHttp:48] HTTP protocol exception (Code=Forbidden).
1:12:24.0: Warning: 0 : [IoManager:48] HTTP error (Forbidden) performing I/O operation on provider.
1:12:24.0: Warning: 0 : [IoManager:48] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded

Not sure what to do with this, upload is stuck, logs don't show anything else since...

 

This is an old drive, right? Eg, created on a much earlier version of StableBit CloudDrive? 

 

If so, try detaching and reattaching the drive. When reattaching the drive, click on "Advanced settings" and set the "Minimum download size" to 10MB, or higher. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#39 Datahoarder

Datahoarder

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 20 February 2017 - 10:26 AM

 

This is an old drive, right? Eg, created on a much earlier version of StableBit CloudDrive? 

 

 

Nope, I started testing CloudDrive on .802 so it was "brand new" drive. Before I saw the newer release I did try reauthorizing, detaching wasn't possible because of files waiting to upload (possibly "force detach" could help, or break it, didn't feel lucky there...).

But as the new update required a restart of the system anyway, it all started working again.



#40 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 20 February 2017 - 09:14 PM

Okay.

 

If the issue continues to occur, please do let us know.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.






Also tagged with one or more of these keywords: ratelimit, google drive, api, exponential backoff

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users