Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

[.901] Google Drive - server side disconnects/throttling


  • Please log in to reply
35 replies to this topic

#1 borez

borez

    Member

  • Members
  • PipPip
  • 20 posts

Posted 09 August 2017 - 12:23 AM

Have been getting lots of throttling/server side disconnect messages from Google Drive recently, see below. Have been using the same settings for some time, seems that there's some changes to their throttling mechanism?

 

If this helps, some comments:

 

1) I had an unsafe shutdown, and as a result i need to reupload 18GB of data. From the tech page, it seems that I'm redownload chunks for partial re-writes. Seems to be fairly intensive vs a straight-upload?

2) The errors have started since I upgraded to the .900/901 builds.

3) I typically run 6 threads, with a max upload speed of 400mbps. Scaling this down to 3/4 threads doesn't help.

0:10:29.5: Warning: 0 : [IoManager:32] Error performing read-modify-write I/O operation on provider. Retrying. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.5: Warning: 0 : [ReadModifyWriteRecoveryImplementation:62] [W] Failed write (Chunk:564, Offset:0x00000000 Length:0x01400500). Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [TemporaryWritesImplementation:62] Error performing read-modify-write, marking as failed (Chunk=564, Offset=0x00000000, Length=0x01400500). Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [WholeChunkIoImplementation:62] Error on write when performing master partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [WholeChunkIoImplementation:62] Error when performing master partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [WholeChunkIoImplementation:61] Error on write when performing shared partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [IoManager:62] Error performing read-modify-write I/O operation on provider. Retrying. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:15:35.9: Warning: 0 : [IoManager:61] Error performing read-modify-write I/O operation on provider. Retrying. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.3: Warning: 0 : [ReadModifyWriteRecoveryImplementation:95] [W] Failed write (Chunk:2743, Offset:0x00000000 Length:0x01400500). Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [TemporaryWritesImplementation:95] Error performing read-modify-write, marking as failed (Chunk=2743, Offset=0x00000000, Length=0x01400500). Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [WholeChunkIoImplementation:95] Error on write when performing master partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [WholeChunkIoImplementation:95] Error when performing master partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [WholeChunkIoImplementation:96] Error on write when performing shared partial write. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [IoManager:95] Error performing read-modify-write I/O operation on provider. Retrying. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:51.6: Warning: 0 : [IoManager:96] Error performing read-modify-write I/O operation on provider. Retrying. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
0:16:56.1: Warning: 0 : [ApiGoogleDrive:102] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:16:56.1: Warning: 0 : [ApiHttp:102] HTTP protocol exception (Code=ServiceUnavailable).
0:16:56.1: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,517ms and retrying.
0:16:56.2: Warning: 0 : [ApiGoogleDrive:79] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:16:56.2: Warning: 0 : [ApiHttp:79] HTTP protocol exception (Code=ServiceUnavailable).
0:16:56.2: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,873ms and retrying.
0:16:56.3: Warning: 0 : [ApiGoogleDrive:100] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:16:56.3: Warning: 0 : [ApiHttp:100] HTTP protocol exception (Code=ServiceUnavailable).
0:16:56.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,561ms and retrying.
0:16:56.3: Warning: 0 : [ApiGoogleDrive:84] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:16:56.3: Warning: 0 : [ApiHttp:84] HTTP protocol exception (Code=ServiceUnavailable).
0:16:56.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,664ms and retrying.
0:16:57.3: Warning: 0 : [ApiGoogleDrive:96] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:16:57.3: Warning: 0 : [ApiHttp:96] HTTP protocol exception (Code=ServiceUnavailable).
0:16:57.3: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,546ms and retrying.


#2 JohnKimble

JohnKimble

    Member

  • Members
  • PipPip
  • 16 posts

Posted 09 August 2017 - 12:11 PM

Been getting those since this morning as-well. Basically started when Windows decided to reboot for an update and my drive got closed non-cleanly and started re-upping my entire cache again. Its getting flooded with these:

Exception:

CloudDriveService.Cloud.Providers.Apis.GoogleDrive.GoogleDriveHttpProtocolException: User rate limit exceeded. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.GetResponse()
   at CloudDriveService.Cloud.Providers.Apis.Base.Parts.HttpApi.HttpApiBase.GetJsonResponse[T](HttpWebRequest request)
   at CloudDriveService.Cloud.Providers.Apis.GoogleDrive.Files.<>c__DisplayClass14_0.<UploadNew>b__1(HttpWebRequest request)
   at CloudDriveService.Cloud.Providers.Apis.Base.Parts.HttpApi.OAuth2HttpApiBase`1.<>c__DisplayClass6_0`1.<RequestBlock>b__0(HttpWebRequest request)
   at CloudDriveService.Cloud.Providers.Apis.Base.Parts.HttpApi.HttpApiBase.<>c__DisplayClass19_0`1.<RequestBlock>b__0()
   --- End of inner exception stack trace ---
   at CoveUtil.RetryBlock.Run[TException,TResult](Func`2 Func, Action`1 Control, Boolean ThrowOnOperationCanceled)
   at CoveUtil.RetryBlock.Run[TException,TResult](Func`1 Func, Action`1 Control)
   at CloudDriveService.Cloud.Providers.Apis.Base.Parts.HttpApi.HttpApiBase.RequestBlock[TResponse](String fullUrl, Func`2 func)
   at CloudDriveService.Cloud.Providers.Apis.Base.Parts.HttpApi.OAuth2HttpApiBase`1.RequestBlock[TResponse](String fullUrl, Func`2 func)
   at CloudDriveService.Cloud.Providers.Apis.GoogleDrive.Files.UploadNew(String fileName, Stream buffer, String parentFolderId, Action`1 generatedId)
   at CloudDriveService.Cloud.Providers.Io.GoogleDrive.GoogleDriveIoProvider.#bjg(#UBf #rxd, ChunkInfo #fAe, #Xtf #gb)
   at CloudDriveService.Cloud.Providers.Io.ChunkIo.ChunkIdIoProvider`2.#4af.#8tf(#UBf #rxd)
   at CloudDriveService.Cloud.Providers.Io.ChunkIo.Helpers.ChunkId.ChunkIdHelper.#Gqe(UInt64 #HAe, Action`1 #wBf)
   at CloudDriveService.Cloud.Providers.Io.ChunkIo.ChunkIdIoProvider`2.#Gqe(ChunkInfo #fAe, #Xtf #gb)
   at #Iqe.#PAe.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#UEf.#Saf.#9Lf(Stream #oYb)
   at #AKf.#zKf.#yKf(ChunkInfo #fAe, Stream #gb, Action`1 #wBf)
   at #Iqe.#UEf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#9Qg.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#6Uf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#3Pf.#jbf.#9Lf(Stream #oYb)
   at #AKf.#zKf.#yKf(ChunkInfo #fAe, Stream #gb, Action`1 #wBf)
   at #Iqe.#3Pf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#NRf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#LCf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#1Sf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hzf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#CAe.#Saf.#9Lf(Stream #oYb)
   at #AKf.#zKf.#yKf(ChunkInfo #fAe, Stream #gb, Action`1 #wBf)
   at #Iqe.#CAe.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#vzf.#U9e.#9Lf(Stream #EQf)
   at #AKf.#zKf.#yKf(ChunkInfo #fAe, Stream #gb, Action`1 #wBf)
   at #Iqe.#vzf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#UAe.#adf.#9Lf(Stream #oYb)
   at #AKf.#zKf.#yKf(ChunkInfo #fAe, Stream #gb, Action`1 #wBf)
   at #Iqe.#UAe.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hzf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hzf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#TEf.#OLf(ChunkInfo #fAe, Stream #gb)
   at #Iqe.#Hqe.#Gqe(ChunkInfo #fAe, Stream #gb)
   at #Kqe.#Tqe.#Wc(UInt64 #Y, Int32 #59f, Stream #vAe, IoType #NOe)
   at CloudDriveService.Cloud.IoManager.#Wc(WriteRequest #89f, RetryInformation`1 #7tf)
   at CloudDriveService.Cloud.IoManager.#iTf.#3bf(RetryInformation`1 #7tf)
   at CoveUtil.RetryBlock..(RetryInformation`1 )
   at CoveUtil.RetryBlock.Run[TException,TResult](Func`2 Func, Action`1 Control, Boolean ThrowOnOperationCanceled)



#3 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 09 August 2017 - 10:30 PM

+1 been getting the same errors the past couple days.

 

2:26:57.5: Warning: 0 : [ApiHttp] Server is throttling us, waiting 4,029ms and retrying.
2:27:02.6: Warning: 0 : [ApiGoogleDrive:143] Google Drive returned error (userRateLimitExceeded): User rate limit exceeded.
2:27:02.6: Warning: 0 : [ApiHttp:143] HTTP protocol exception (Code=ServiceUnavailable).
2:27:02.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 8,437ms and retrying.


#4 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 10 August 2017 - 12:38 AM

I am also seeing tons of errors. Perhaps google made a server-side change here?



#5 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 10 August 2017 - 04:59 PM

looks like rclone is experiencing similar issues.. https://forum.rclone...xceeded/3469/17

 

hope this isn't the beginning of the end.



#6 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 10 August 2017 - 05:28 PM

I mean, the good news is that the drives are still working. Or, at least, mine are. But maybe there are going to be additional throttling concerns. 

 

Anyone noticed if it's on reads and writes? As far as I can tell it's only writes, no?



#7 janchrst

janchrst

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 10 August 2017 - 05:30 PM

I mean, the good news is that the drives are still working. Or, at least, mine are. But maybe there are going to be additional throttling concerns.

Anyone noticed if it's on reads and writes? As far as I can tell it's only writes, no?


Thats my exp. as well.


Sent from my iPhone using Tapatalk

#8 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 10 August 2017 - 06:13 PM

Reading through the rclone thread and some other forums, it looks like it might just be a new upload limit. I'm reading a lot of people say that it kicks in at around 800GB/day or so. That's a pretty reasonable limit imo, but it's probably going to change some things for CloudDrive and other solutions. We'll have to see what Christopher has to say when he has a chance to hop on the forums.

 

Sadly, for me, I was in the middle of transferring data from one large drive to two smaller ones (to delete the large one), so that's going to take a lot longer now. 



#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 11 August 2017 - 02:16 AM

the error in question is a throttling related error, and I believe is account wide, not user based.

 

Try reauthorizing the drive, as this may help.

 

 

Also, this won't go away right away when upgrading, it make take a bit before it starts working properly again.

 

And the 901 build should significantly reduce the amount of API calls, and the amount of data requested.  It does this by changing the "minimum download size" code, and making it more intelligent and efficient.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 11 August 2017 - 05:04 AM

Oh, oops, I'm on the latest stable. I'll upgrade to 901 beta and see if the problem is alleviated.



#11 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 11 August 2017 - 05:14 AM

It seemed to fix it for about 5 minutes and then went back to:

 

0:04:22.1: Warning: 0 : [ApiGoogleDrive:8] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:04:22.1: Warning: 0 : [ApiHttp:8] HTTP protocol exception (Code=ServiceUnavailable).
0:04:22.1: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,294ms and retrying.
0:04:22.6: Warning: 0 : [ApiGoogleDrive:57] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded
0:04:22.6: Warning: 0 : [ApiHttp:57] HTTP protocol exception (Code=ServiceUnavailable).
0:04:22.6: Warning: 0 : [ApiHttp] Server is throttling us, waiting 1,075ms and retrying.
0:04:32.9: Warning: 0 : [OAuth2TokenBase:11357795] HttpProtocolException refreshing token. HTTP Status BadRequest
0:04:32.9: Warning: 0 : [ValidateLogins] Error instantiating management provider. Security error.


#12 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 11 August 2017 - 06:01 AM

To be clear: This is not going to go away by upgrading. The cutoff is server side and transfer based. Once you upload somewhere around 750-800GB of data in a 24 hour period you will be API locked until the following day. CloudDrive can certainly add some features to help us manage that total, but the throttling will not go away unless Google decides to change their policies again. 

 

This simply appears to be a new limitation being imposed by Google. Uploads are now limited, server-side, to around 750GB or so per day. As far as anyone can tell, reads are not limited--they seem to function as they always have. 



#13 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 11 August 2017 - 10:24 AM

To be clear: This is not going to go away by upgrading. The cutoff is server side and transfer based. Once you upload somewhere around 750-800GB of data in a 24 hour period you will be API locked until the following day. CloudDrive can certainly add some features to help us manage that total, but the throttling will not go away unless Google decides to change their policies again. 

 

This simply appears to be a new limitation being imposed by Google. Uploads are now limited, server-side, to around 750GB or so per day. As far as anyone can tell, reads are not limited--they seem to function as they always have. 

 

 

Correct.

 

Once the issue is triggered, only time will fix it.

 

 

That said, the newer version should help with avoiding hitting this issue, as it should be more efficient about using what has been downloaded and keeping it in the cache better.   This should reduce the likelihood that it will trigger this.

 

 

that said .... it may be a good idea to throttle your speeds....  setting it to 50/25 mbps  should allow you to (narrowly) avoid the 800-900GB/day apparent limit.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#14 borez

borez

    Member

  • Members
  • PipPip
  • 20 posts

Posted 12 August 2017 - 06:59 AM

BTW I'm hitting the errors upon startup, so it seems to be independent on upload caps. And I've only managed to upload at speeds < 200mbps.

 

So something has changed with Google, but without a clear trend.



#15 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 12 August 2017 - 10:55 AM

Are you sure it's the "User rate limit exceeded" issue?

 

If it is, then ... that will be normal, as it takes time for this status to be removed from your account.  

 

 

And for reference, 75mbps for 24 hours will hit around 800-900GBs.  So that may/will hit this limit.  It's ... disturbingly low, IMO. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#16 JohnKimble

JohnKimble

    Member

  • Members
  • PipPip
  • 16 posts

Posted 12 August 2017 - 02:43 PM

 

I ask him if there is a limit for uploading XXX GB per 24 hours. This is the answer: ā€œI have double checked and there is no limitations regarding that, as per this article : https://support.goog...2759172849&rd=1ā€



#17 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 12 August 2017 - 06:55 PM

 

 

I ask him if there is a limit for uploading XXX GB per 24 hours. This is the answer: ā€œI have double checked and there is no limitations regarding that, as per this article : https://support.goog...2759172849&rd=1ā€

 

 

Google does not publish their API limits. So you won't see a public statement on this. Many people have tested it though. The limit is around 750GB/day. This is the same way we discovered the API limits that rClone previously ran into as well. You can always test it yourself if you want to verify. Stop uploading data for at least 24 hours and then track how much you upload before you start getting 403 errors. 



#18 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 12 August 2017 - 07:45 PM

Google does not publish their API limits. So you won't see a public statement on this. Many people have tested it though. The limit is around 750GB/day. This is the same way we discovered the API limits that rClone previously ran into as well. You can always test it yourself if you want to verify. Stop uploading data for at least 24 hours and then track how much you upload before you start getting 403 errors. 

 

 

Sadly, that's not entirely correct.

 

Google has been very clear with their published API limits, in how many requests in a day, etc.

 

This new behavior ... is well new, and doesn't appear to be documented.  This is definitely not "typical google behavior" and doesn't bode well for consumers....  (or us, for that matter)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#19 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 12 August 2017 - 10:26 PM

I just know that there is an unpublished 10TB/day download limit as well, and limits on drive to drive transfers. I know they publish other details, but it seems like their fair use policies (if you want to call them that) are more obscure for some reason. Their own support folks don't even seem to be aware. It is, in any case, a concerning practice.



#20 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 13 August 2017 - 12:39 AM

Maybe, but 10TB/day is a huge amount, and most won't hit that. In fact, you'd need near gigabit internet to hit that. 

 

But that's not really the point hear.

 

As for the "fair use", the blunt term will be "bandwidth cap", Because that is what it is.  It may be rated per day, but at the end of the day, that's what this is.

 

And it sucks, all around.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users