Poor performance Google Drive (not rate limit)

[Joel Kåberg 6]

I'm seeing poor upload performance with GD, it goes somewhere between 0 to 150 mbps (on average 50-70mbps) but very rarely above that.

Here's my settings https://goo.gl/iXdlWc | https://goo.gl/H5T8G7

 

An regular Speedtest https://goo.gl/iRjC8o

 

Looking over the logs I can't see any Rate limit or similar, just a bunch of Write chunk

 

 

14:48:50.5: Information: 0 : [GoogleDriveIoProvider:39] [W] Writing chunk 37,743.

14:48:51.1: Information: 0 : [GoogleDriveIoProvider:29] [W] Writing chunk 37,744.

14:48:51.9: Information: 0 : [GoogleDriveIoProvider:64] [W] Writing chunk 37,745.

14:48:53.3: Information: 0 : [GoogleDriveIoProvider:14] [W] Writing chunk 37,746.

14:48:54.8: Information: 0 : [GoogleDriveIoProvider:45] [W] Writing chunk 37,747.

14:48:57.1: Information: 0 : [GoogleDriveIoProvider:41] [W] Writing chunk 37,748.

 

My workflow

Ubuntu server with samba (main storage for now)->1gbit lan->Windows 10 PC with clouddrive->google drive

 

So I take it this is something related to my ISP or something between me and Google?

 

I'm investigateing now if my data gets uploaded to EU or US datacenters (I'm located in the EU) Uploaded to EU/Amsterdam

 

For comparision I can saturate my bandwith with the same Google Drive using rlcone, so this must be a CloudDrive issue.

 

One thing that I've noticed is when I pause the transfer to the CloudDrive (and Clouddrive just sits uploading) I'm seeing a lot better bandwith usage (nearly 100%). I've now tried several (SSD) disk on different PC but they all suffer the same fate.

So I think I got this down to IO issues, it seems better without encryption but I'd love haveing that. Task manager gives a pretty good hint as to the disk beeing excessevly used (I'm assuming due to the cache in CloudDrive). Perhaps this need some attention, also an RAM option would be nice here (for us with plenty of it). I tried creating a RAMdrive to see if that helped (use as a cache drive) but CloudDrive won't recognize it

[Christopher (Drashna) 631]

If pausing the connection and then restarting it fixes the issue, then the issue is likely a throttling issue.  More than likely, we're getting throttling requests for Google and respecting it. 

 

And if this is the case, try reducing the number of threads that Google Drive is using.  And if possible, try using larger chunk sizes when creating the drive, and see if that helps. 

[Joel Kåberg 6]

Not pausing the connection, pausing the transfer to the (local) CloudDrive (while CloudDrive tries to upload)

 

What happends is I try to copy stuff to the local CloudDrive, CloudDrive then encrypts and uploads to Google Drive. When all this is happeing at the same time I see disk utilization hitting 100% and uploads really seem's to suffer from it as explained above

 

Seem's the way CloudDrive handles disk IO is really killing it performance wise. I've now moved to rclone while this gets sorted (which caches in ram instead of on disk which i assume CloudDrive is doing as ram is hardly used by CloudDrive)

[Joel Kåberg 6]

Just to make my point, here's a video: https://eth0.im/upload/2016-11-17_23-19-10.mp4

 

At 1 min I pause the transfer (from an SMB share) to the CloudDrive (while CD is still uploading to Google Drive)

At 2 min I resume the transfer from SMB to CD

At 3 min I show you system specs

 

Pay attention to the upload speed. Normally it's steady above 300 mbps (while SMB transfer is paused) and occasionally hits 400 mbps, just had a bad run here.

 

So while transfering from SMB to CD I see an 30-40% performance loss uploading to Google Drive, and I suspect/guess this is due to CD beeing dependent on the OS drive to split up chunks etc(?). So my suggestion is to offload I/O heavy actions into RAM (atleast make it optional), which should speed things up nicely.

[wid_sbdp 3]

Is the API issue being worked at all with Google yet? Last I read it's "something we're going to address with them." I'm super pro-StableBit, but I'm honestly about to give up and just go to massive amounts of unencrypted drives pooled together and use something like rclone and risk the ToS bans because it would be 100000x quicker to reupload everything at 500 MB/s than it would encrypted at 11 MB/s :/

[Christopher (Drashna) 631]

Is the API issue being worked at all with Google yet? Last I read it's "something we're going to address with them." I'm super pro-StableBit, but I'm honestly about to give up and just go to massive amounts of unencrypted drives pooled together and use something like rclone and risk the ToS bans because it would be 100000x quicker to reupload everything at 500 MB/s than it would encrypted at 11 MB/s :/

 

In the latest version (770), we've implemented an API pool, so that may help with performance. 

 

You will need to re-authorize the drives to use this pool.  This should be found under "Disk Options" for each drive. 

 

If you have done this and you're still seeing performance issues, let me know.

[Joel Kåberg 6]

Yes I noticed very bad performance (not even close to the BW usage you see in my video) with the old keys (roughly around 20~60 mbps) so I updated on the 20th to the 770 build. Initial speeds was OK for a few hours (5-6), around 200~280 mbps with 100% I/O usage.

 

Yesterday speeds went back to the same as with the old api keys (40-80 mbps) and has been steady since then. I/O usage on the HDD is now at 30~50% so I'm sure it isn't that.

 

If Alex want's I'd let him remote controll the PC doing the upload, if he want's to debug.

PS: yes, I did re-authorize.

[Christopher (Drashna) 631]

Yes I noticed very bad performance (not even close to the BW usage you see in my video) with the old keys (roughly around 20~60 mbps) so I updated on the 20th to the 770 build. Initial speeds was OK for a few hours (5-6), around 200~280 mbps with 100% I/O usage.

 

Yesterday speeds went back to the same as with the old api keys (40-80 mbps) and has been steady since then. I/O usage on the HDD is now at 30~50% so I'm sure it isn't that.

 

If Alex want's I'd let him remote controll the PC doing the upload, if he want's to debug.

 

PS: yes, I did re-authorize.

 

 

Could you grab the logs from the system? 

 

And do you see a "turtle" icon next to the bandwidth usage? 

 

If so, then the slower speed may be because you're being throttled.  And in this case, reducing the number of threads may help.

[thnz 13]

In the latest version (770), we've implemented an API pool, so that may help with performance. 

 

Hopefully Google doesn't take exception to that.

[jimbb 0]

I experience the same issue as well and on the latest version (770).

 

I can upload between 300-600 mbps on every fresh reboot, and after a while, it drops to 50-150 mbps and later dropped to 0-50 mbps. (On a 1Gbps line)

 

The weird thing is that on every fresh restart, I can go back to 300-600 mbps for a while until it drops again. From the look of it, it seems more like a CloudDrive issue rather than Google.

 

Tried changing the thread settings etc and nothing seems to help.

 

Already reported to support and waiting for a resolution

[Joel Kåberg 6]

I experience the same issue as well and on the latest version (770).

 

I can upload between 300-600 mbps on every fresh reboot, and after a while, it drops to 50-150 mbps and later dropped to 0-50 mbps. (On a 1Gbps line)

 

The weird thing is that on every fresh restart, I can go back to 300-600 mbps for a while until it drops again. From the look of it, it seems more like a CloudDrive issue rather than Google.

 

Tried changing the thread settings etc and nothing seems to help.

 

Already reported to support and waiting for a resolution

 

Exactly the same issue for me (just lower bandwidth  ), I'll see if I can find some logs when I get home (OS logs or CloudDrive?)

[raidz 4]

I am seeing the exact same issues as these two guys. I uploaded logs about a week ago so hopefully Alex can give this some time soon.

[jimbb 0]

Exactly the same issue for me (just lower bandwidth  ), I'll see if I can find some logs when I get home (OS logs or CloudDrive?)

 

I am seeing the exact same issues as these two guys. I uploaded logs about a week ago so hopefully Alex can give this some time soon.

 

 

They came back to me this morning saying it might be Google drive who is limiting the API request frequent. However, I disagree as I have set the thread number to 1 on both upload and download and the speed drop is still significant comparing between fresh reboot (50-100mbps per thread) vs after 2-3 hours of run time (0-15 mbps per thread).

 

 They have advised me to tune up the chunk size as well which I created a new drive to test the theory, but still having the same issue over time

 

Will keep you guys as we go along. 

 

[raidz 4]

My Chunk size is at 100mb, Minimum download at 10mb, Download threads at 12 and Upload Threads at 6. No matter how many threads I set (more or less) it is still slow as molasses (on a 1gbit/1gbit line).  Download speeds have decreased as well due to delay in thread response time. This started happening to me after .726 and has not been different since. 

 

I do believe this might has to do with Google limiting API request but also think it is something they changed in the software that is triggering it unless Google just made a big change to how they handle requests.

[Christopher (Drashna) 631]

Hopefully Google doesn't take exception to that.

 

Hopefully.

 

But we're not round-robining them API per request.  The re-authorization process picks one "at random", and the drive uses that key for the entire time it's mounted. 

 

If there is an issue with this behavior, we'll deal with at that time (or before, because yeah, I'm not a fan of it either)

 

 

 

 

They came back to me this morning saying it might be Google drive who is limiting the API request frequent. However, I disagree as I have set the thread number to 1 on both upload and download and the speed drop is still significant comparing between fresh reboot (50-100mbps per thread) vs after 2-3 hours of run time (0-15 mbps per thread).

 

 They have advised me to tune up the chunk size as well which I created a new drive to test the theory, but still having the same issue over time

 

Will keep you guys as we go along. 

 

Checking the text logs can verify that.  If you're seeing a lot of throttling requests, then that's what going on.

 

Otherwise, logging the traffic with FiddlerCap (http://www.telerik.com/fiddler/fiddlercap), usin the "decrypt https traffic" option may help us to identify the issue.

[Joel Kåberg 6]

Just thought I'd say this is still happening on .776 (initial speed is very good, after an hour or two it drops significantly). No indication in the text logs to as if this is rate or throttling of any kind.

 

This is on a brand new drive, created for the very purpose of testing this issue.

 

Settings: https://goo.gl/tp4ngZ

[steffenmand 60]

It's the http response time that seems to increase over time - also happens here and nothing in the logs. A fresh reboot makes it all fine again! Takes some days for it to get really bad here though

[thnz 13]

Perhaps the exponential backoff isn't resetting itself or something.

[Christopher (Drashna) 631]

Hmmm,  on an offchance, have you ran the network auto-tuning command? 

 

Specifically, try running the following from an administrative command line, and reboot.

 

netsh int tcp set global autotuninglevel=highlyrestricted

[Joel Kåberg 6]

Hmmm,  on an offchance, have you ran the network auto-tuning command? 

 

Specifically, try running the following from an administrative command line, and reboot.

 

 

Yeah, I applied your suggested fix last night. Still same thing :-/

[Gbyrd 7]

I have not been able to get Google drive working well. The download is always limited to about 3-4 Mbps. Before it use to be much higher, seems like the more people using this app the performance takes a heavy hit because of it. Anyway to get it resolved?

[Christopher (Drashna) 631]

I have not been able to get Google drive working well. The download is always limited to about 3-4 Mbps. Before it use to be much higher, seems like the more people using this app the performance takes a heavy hit because of it. Anyway to get it resolved?

The latest versions ( 1.0.0.770+ ) use a pool of API keys.  If you've installed that version (ideally the 1.0.0.777) build and have re-authorized the drive, it should be using a random key.  This should significantly reduce the "rate limit exceeded" issue, if not eliminate it entirely (a main cause of the performance issues). 

 

Additionally, try re-attaching the drive, and increase the "minimum download size". 

Otherwise, try creating a new driver and use larger chunk sizes, as well. 

[Joel Kåberg 6]

The latest versions ( 1.0.0.770+ ) use a pool of API keys.  If you've installed that version (ideally the 1.0.0.777) build and have re-authorized the drive, it should be using a random key.  This should significantly reduce the "rate limit exceeded" issue, if not eliminate it entirely (a main cause of the performance issues). 

 

Additionally, try re-attaching the drive, and increase the "minimum download size". 

Otherwise, try creating a new driver and use larger chunk sizes, as well. 

 

As you know I've already tried all this, and I'm on the very latest BETA build. The issue essentially boils down to the response time which increases over time, this really kills the performance of CD.

[wid_sbdp 3]

I don't think it's CD specifically, I think it's the interaction with ACD and Google. I made a new Azure account (with the free $200 in credit you get when you make one) and built a storage instance, added that to CD and uploaded some stuff. I was getting very fast 700mbit transfers to it. I think ACD and GCD are just not "as open" as the respective enterprise services are in terms of speed and capability (number of connections, etc). I mean it would make sense to limit those on Amazon's and Google's side as they are cheaper alternatives to paying for an enterprise-level tier.

 

Which, is fine by me. Stability + 150mbit uploads to ACD for $5/mo beats out Stability + 700mbit uploads for $200-300+ (and into the thousands and tens of thousands if you're crazy and have 200TB on your drive).

[Gbyrd 7]

The latest versions ( 1.0.0.770+ ) use a pool of API keys.  If you've installed that version (ideally the 1.0.0.777) build and have re-authorized the drive, it should be using a random key.  This should significantly reduce the "rate limit exceeded" issue, if not eliminate it entirely (a main cause of the performance issues). 

 

Additionally, try re-attaching the drive, and increase the "minimum download size". 

Otherwise, try creating a new driver and use larger chunk sizes, as well. 

 

After reattaching it works perfectly again. I didn't change the minimum download size however its still set to off.  After trying it again it seems to be downloading at a lower 15 mbps, ive seen it reach up to 50mbps. My line is 1gbps down. After the prefetch it works well, but the initial part is a bit tough.

[raidz 4]

This issue is pretty much making cd unusable after a few hours for me at this point. Restarting every few hours is a bad solution.

 

Any progress being made towards a fix on this Christopher?