Problems with Google Drive in Hetzner server

[Chupa]

Salve, oggi ho problemi con Google Drive non riesco a caricare e scaricare mi da anche errori. L'upload non sembra totalmente bloccato, la quota di cache sta diminuendo molto lentamente ma continuo a ricevere messaggi di errore e non voglio corrompere il disco rischiando di doverlo distruggere e ricostruirlo come nuovo. Vi lascio il log così potete capire meglio

 

Citazione

Avviso CloudDrive.Service.exe 0 [IoManager:18] Errore durante l'esecuzione dell'operazione di I/O di lettura sul provider. Nuovo tentativo. Stato HTTP BadGateway 2023-01-28 14:04:59Z
2122477680370 Avviso CloudDrive.Service.exe 0 [ApiGoogleDrive:186] Errore restituito da Google Drive (notFound): File non trovato: 17t1DKD-5t_vuoKlt9RZEeVvm61uc9fTq. 2023-01-28 14:04:59Z 2122478886991
CloudDrive.Service.exe Avviso 0 [ApiHttp:186] Eccezione protocollo HTTP (Code=NotFound). 2023-01-28 14:04:59Z 2122478887994
CloudDrive.Service.exe Avviso 0 [IoManager:186] Errore durante l'esecuzione dell'operazione di I/O di scrittura sul provider. Nuovo tentativo. Impossibile scrivere i dati nella connessione di trasporto: una connessione esistente è stata forzatamente chiusa dall'host remoto. 2023-01-28 14:04:59Z 2122479113130
CloudDrive.Service.exe Avviso 0 [ApiGoogleDrive:193] Google Drive ha restituito un errore: [impossibile analizzare] 2023-01-28 14:04:59Z 2122479215378
CloudDrive.Service.exe Avviso 0 [ApiHttp:193] Eccezione protocollo HTTP (codice =BadGateway). 2023-01-28 14:04:59Z 2122479216067
CloudDrive.Service.exe Avviso 0 [IoManager:193] Errore HTTP (BadGateway) durante l'esecuzione dell'operazione di I/O di scrittura sul provider. 2023-01-28 14:04:59Z 2122479268039

 

Continuo a ricevere questi errori Ho anche provato ad autorizzare di nuovo ma non ha aiutato, e provando a riavviare il pc, inizialmente non riusciva a montare i dischi.

 

Grazie

 

Ho dimenticato di aggiungere che la mia versione è la 1.2.3.1633 e utilizzo un server Hetzner

REGISTRO .txt

[Chupa]

Hello, I wanted to report that the problem is solved, simply set Cloudfare as DNS servers then set 1.1.1.1 as primary server and Google 8.8.8.8 as secondary, this problem is limited to Hetzner servers. I have reported the issue to their support and they are looking into fixing it.

[Jellepepe]

Thank you!!!
 

I was having this same exact error, though intermittently (about 10-20% of requests were going through without issue) so I did not suspect DNS to be the culprit.
I also already had cloudflare as my secondary DNS server so I assumed any dns failures would be covered. Access to google drive through other tools seemed unaffected. I suppose the hetzner dns is not failing the dns but somehow providing stale data for the specific api endpoint clouddrive is using.

For future reference I also needed to flush my dns cache for clouddrive to start working again, this is done using the command:
 

ipconfig /flushdns

in an (elevated) command prompt window.

I will also submit a ticket to Hetzner support to make them aware of the issue (my server is in FSN, is that the case for yours as well? @Chupa)
I suggest in the meantime perhaps renaming this topic to include hetzner in the title so others with the same issue can find it easily

[Chupa]

Hello, you are right in changing the title of the topic, and so I did, thanks for the suggestion.

My server is in Falkenstein, I've already opened a ticket with Hetzner, but an extra feedback certainly doesn't hurt.

Have a good evening

[Vexus]

On 1/29/2023 at 7:22 PM, Chupa said:

Hello, you are right in changing the title of the topic, and so I did, thanks for the suggestion.

My server is in Falkenstein, I've already opened a ticket with Hetzner, but an extra feedback certainly doesn't hurt.

Have a good evening

Hi, i have the same problem, was the Hetzner support any help? thanks

[Jellepepe]

On 2/2/2023 at 11:04 AM, Vexus said:

Hi, i have the same problem, was the Hetzner support any help? thanks

The issue was resolved for me by changing the dns provider to either cloudflare or google and flushing the dns cache as described above, this will likely work for you too.

I contacted hetzner support to notify them of the issue and they indicated that they could not detect any issues with the dns server (which makes sense since it is not a resolving issue, the issues is perhaps that it is resolving to rate limited servers leading to the errors we are seeing, which is hard to test for)

They also mentioned that they were still looking into the issue and were contacting google to see if they can find the issue.

My personal guess is that the Hetzner dns uses a custom dns resolving for google services to optimize their endpoint usage (a lot of major datacenters do this for major providers like google or amazon AWS) - these google drive endpoints seem to be (perhaps erroneously) rate-limiting access, which is leading to the errors we are seeing.

Whether this issue is due to an issue on googles side (erroneous rate limiting) or on hetzners side (stale dns records pointing to incorrect endpoints) is unclear at this point, but I assume the issue will be determined soon once Hetzner gets into contact with google.

For the time being we can just switch to a different dns provider for google drive api access

[Jellepepe]

Well... The issue has reappeared for me - same failures with the same messages.
Dns is already on google dns for this VM so that can't be the issue now.

Do you also experience the same again @Chupa?

If anyone has any insight that would be great

EDIT:
Seems to indeed be the same issue again, manually adding dns records for the www.googleapis.com domain has seemingly resolved the issue.
I assume there is some hetzner customer(s) abusing the google api endpoints causing them to automatically rate limit the hetzner ip ranges, which is also affecting other users - very frustrating.

Not sure there is any way for us to 'fix' this reliably

[steffenmansen]

12 hours ago, Jellepepe said:

Well... The issue has reappeared for me - same failures with the same messages.
Dns is already on google dns for this VM so that can't be the issue now.

Do you also experience the same again @Chupa?

If anyone has any insight that would be great

EDIT:
Seems to indeed be the same issue again, manually adding dns records for the www.googleapis.com domain has seemingly resolved the issue.
I assume there is some hetzner customer(s) abusing the google api endpoints causing them to automatically rate limit the hetzner ip ranges, which is also affecting other users - very frustrating.

Not sure there is any way for us to 'fix' this reliably

Hi,

Can you help explained in detail what you did ?

I constantly have errors and now also get error 429 on top!

[steffenmansen]

I give up 😞

Constant BadGateway errors and my DNS is set to Google's 8.8.8.8 / 8.8.4.4

Have no clue what to do and Hetzner claims its not them and that they have no issues with the google apis

if someone solved this please do provide context as to how you solved it!

[steffenmansen]

A nice guy helped me!

Simply add this to your hosts file!

142.250.113.95 www.googleapis.com
142.251.16.95 www.googleapis.com
172.253.63.95 www.googleapis.com
142.251.111.95 www.googleapis.com
142.251.163.95 www.googleapis.com
142.251.167.95 www.googleapis.com
142.250.74.78 www.googleapis.com

[CokeZero]

On 3/13/2023 at 9:04 PM, steffenmansen said:

A nice guy helped me!

Simply add this to your hosts file!

142.250.113.95 www.googleapis.com
142.251.16.95 www.googleapis.com
172.253.63.95 www.googleapis.com
142.251.111.95 www.googleapis.com
142.251.163.95 www.googleapis.com
142.251.167.95 www.googleapis.com
142.250.74.78 www.googleapis.com

By Me make this a very bad Upload Speed. If i delete this the Speed is normal, if i add this speed is very slow.

Not Understand why, by a friend he have no Problems with this.

[Jellepepe]

On 3/12/2023 at 12:56 PM, steffenmansen said:

Hi,

Can you help explained in detail what you did ?

I constantly have errors and now also get error 429 on top!

Right, as I mentioned the errors are the same as before, the 429 error is an endpoint rate limit error. The endpoints that are returned by the dns (at first only Hetzners own dns, but it seems now also the ones returned by Google and Cloudflare dns. What exactly is the cause of these endpoint rate limits is unclear though, presumably something on the Hetzner network is producing an abusive amount of requests which results in the endpoints rate limiting the entire Hetzner ip ranges.

@Christopher (Drashna) Perhaps it would  be possible to add a slightly more comprehensive error handling for 429 errors for the google drive provider? E.g. a clearer error message and more aggressive backoff behaviour (perhaps limit the entire provider connection to 1-2 threads & disable auto-remount) It seems the current implementation doesn't deal with this situation super well as a small amount of requests do still succeed. I am seeing some filesystem corruption (luckily recoverable) due to my drives brute force trying to get through the throttling almost a full day before I was able to manually unmount them & diagnose the issue.

On 3/13/2023 at 9:04 PM, steffenmansen said:

A nice guy helped me!

Simply add this to your hosts file!

142.250.113.95 www.googleapis.com
142.251.16.95 www.googleapis.com
172.253.63.95 www.googleapis.com
142.251.111.95 www.googleapis.com
142.251.163.95 www.googleapis.com
142.251.167.95 www.googleapis.com
142.250.74.78 www.googleapis.com

Right, this is what I mentioned in the edit to my post. Manually adding dns records to other endpoints that are not rate limiting is a temporary workaround.
This is definitely not ideal though as when not relying on dns there is no way to guarantee these won't become stale & these won't be the closest endpoints to begin with. Let alone the original issue of whatever is causing the rate limiting to begin with may very well appear here as that service/actor may also start abusing other endpoints

54 minutes ago, CokeZero said:

By Me make this a very bad Upload Speed. If i delete this the Speed is normal, if i add this speed is very slow.

Why are you trying to apply this if the speed is normal without adding these records? this is a temporary workaround for a rate limit error, the speeds I'm seeing with the rate limit active is a few kb/s at most (and 90%+ failures)
The speed being lower is normal as this is manually targeting the 'wrong' google api endpoints, which are likely not very close / connected ideally to the Hetzner data center.
 

1 hour ago, CokeZero said:

Not Understand why, by a friend he have no Problems with this.

Your friend likely isn't using a Hetzner server or at least not at the same location. I'm only seeing the error in Hetzner Falkenstein (FSN1), I'm not sure if its only that datacenter or the entire Hetzner ip ranges.

There's not really any way for us to actively fix this issue. Hetzner and/or Google are the only ones able to identify the cause of the rate limiting and stop it, and they are likely both going to blame the other party and say they are the ones that should fix it. We can try opening a ticket with either/both but I'm not very hopeful.

[CokeZero]

On 3/15/2023 at 1:07 AM, Jellepepe said:

Why are you trying to apply this if the speed is normal without adding these records?

I have a Hetzner NVME Server, and i have the Problems with Google Cloud what all Users have momently. I use this Hits from Here and add is the Ips in the Hostfile the Speed is under 50mbits, somites go 0mbits and go up again , but not over 50mbits. . Delete i the Ips the Speed will go over 200mbits.
I testet this with 30GB Upload. Without the Ips in the Host i become after 2GB Upload the Google Error in Clouddrive. Now i make Upload break, add this in the Host, Upload start, the Upload works for 10GB, but with very slow Speed. break, delete the Ips in the Host, Start, again good Speed but after short Time, Google Error.

So i can find out thats the IPs in the Host make the Speed, whyever to me, slow.

 

On 3/15/2023 at 1:07 AM, Jellepepe said:

Your friend likely isn't using a Hetzner server or at least not at the same location. I'm only seeing the error in Hetzner Falkenstein (FSN1), I'm not sure if its only that datacenter or the entire Hetzner ip ranges.

A Friend from to me have also a Hetzner NVME and the Google Problem with Clouddrive and have no problems with the IPs in the Host.
Only I have the Slow speed only with the IPs in the Host.

[Chupa]

Hi guys, I update everyone with the solution that another user has recommended and that I have adopted.
I took a 1 Gbit/s VPS in another provider (not Hetzner) and I installed a Wireguard server on it, then I installed a Wireguard client on my server where I use Clouddrive and I set it to forward all traffic to and from the intervals IP of Google API in the VPS in this way I have eliminated the errors, I have been using this method for a few weeks and I have not received any more errors.

Side note: When I copy a file from a local disk to a cloud disk the transfer speed that Windows shows me is about 35 MB/s and I don't know if this is a problem related to this solution or to the NVMe disk where the cache resides replaced at the same time as the "Wireguard solution".

I hope I was useful to someone