Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Poor performance Google Drive (not rate limit)


  • Please log in to reply
35 replies to this topic

#1 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 09 November 2016 - 09:45 AM

I'm seeing poor upload performance with GD, it goes somewhere between 0 to 150 mbps (on average 50-70mbps) but very rarely above that.

Here's my settings https://goo.gl/iXdlWc | https://goo.gl/H5T8G7

 

An regular Speedtest https://goo.gl/iRjC8o

 

Looking over the logs I can't see any Rate limit or similar, just a bunch of Write chunk

 

 

14:48:50.5: Information: 0 : [GoogleDriveIoProvider:39] [W] Writing chunk 37,743.

14:48:51.1: Information: 0 : [GoogleDriveIoProvider:29] [W] Writing chunk 37,744.
14:48:51.9: Information: 0 : [GoogleDriveIoProvider:64] [W] Writing chunk 37,745.
14:48:53.3: Information: 0 : [GoogleDriveIoProvider:14] [W] Writing chunk 37,746.
14:48:54.8: Information: 0 : [GoogleDriveIoProvider:45] [W] Writing chunk 37,747.
14:48:57.1: Information: 0 : [GoogleDriveIoProvider:41] [W] Writing chunk 37,748.

 

My workflow

Ubuntu server with samba (main storage for now)->1gbit lan->Windows 10 PC with clouddrive->google drive

 

So I take it this is something related to my ISP or something between me and Google?

 

I'm investigateing now if my data gets uploaded to EU or US datacenters (I'm located in the EU) Uploaded to EU/Amsterdam

 

For comparision I can saturate my bandwith with the same Google Drive using rlcone, so this must be a CloudDrive issue.

 

One thing that I've noticed is when I pause the transfer to the CloudDrive (and Clouddrive just sits uploading) I'm seeing a lot better bandwith usage (nearly 100%). I've now tried several (SSD) disk on different PC but they all suffer the same fate.

So I think I got this down to IO issues, it seems better without encryption but I'd love haveing that. Task manager gives a pretty good hint as to the disk beeing excessevly used (I'm assuming due to the cache in CloudDrive). Perhaps this need some attention, also an RAM option would be nice here (for us with plenty of it). I tried creating a RAMdrive to see if that helped (use as a cache drive) but CloudDrive won't recognize it


  • Ginoliggime, KiaraEvirm and Antoineki like this

#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 09 November 2016 - 06:04 PM

If pausing the connection and then restarting it fixes the issue, then the issue is likely a throttling issue.  More than likely, we're getting throttling requests for Google and respecting it. 

 

And if this is the case, try reducing the number of threads that Google Drive is using.  And if possible, try using larger chunk sizes when creating the drive, and see if that helps. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 13 November 2016 - 11:45 AM

Not pausing the connection, pausing the transfer to the (local) CloudDrive (while CloudDrive tries to upload)

 

What happends is I try to copy stuff to the local CloudDrive, CloudDrive then encrypts and uploads to Google Drive. When all this is happeing at the same time I see disk utilization hitting 100% and uploads really seem's to suffer from it as explained above

 

Seem's the way CloudDrive handles disk IO is really killing it performance wise. I've now moved to rclone while this gets sorted (which caches in ram instead of on disk which i assume CloudDrive is doing as ram is hardly used by CloudDrive)



#4 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 17 November 2016 - 10:33 PM

Just to make my point, here's a video: https://eth0.im/uplo...17_23-19-10.mp4

 

At 1 min I pause the transfer (from an SMB share) to the CloudDrive (while CD is still uploading to Google Drive)

At 2 min I resume the transfer from SMB to CD

At 3 min I show you system specs

 

Pay attention to the upload speed. Normally it's steady above 300 mbps (while SMB transfer is paused) and occasionally hits 400 mbps, just had a bad run here.

 

So while transfering from SMB to CD I see an 30-40% performance loss uploading to Google Drive, and I suspect/guess this is due to CD beeing dependent on the OS drive to split up chunks etc(?). So my suggestion is to offload I/O heavy actions into RAM (atleast make it optional), which should speed things up nicely.



#5 wid_sbdp

wid_sbdp

    Advanced Member

  • Members
  • PipPipPip
  • 49 posts

Posted 19 November 2016 - 04:29 AM

Is the API issue being worked at all with Google yet? Last I read it's "something we're going to address with them." I'm super pro-StableBit, but I'm honestly about to give up and just go to massive amounts of unencrypted drives pooled together and use something like rclone and risk the ToS bans because it would be 100000x quicker to reupload everything at 500 MB/s than it would encrypted at 11 MB/s :/



#6 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 21 November 2016 - 09:00 PM

Is the API issue being worked at all with Google yet? Last I read it's "something we're going to address with them." I'm super pro-StableBit, but I'm honestly about to give up and just go to massive amounts of unencrypted drives pooled together and use something like rclone and risk the ToS bans because it would be 100000x quicker to reupload everything at 500 MB/s than it would encrypted at 11 MB/s :/

 

In the latest version (770), we've implemented an API pool, so that may help with performance. 

 

You will need to re-authorize the drives to use this pool.  This should be found under "Disk Options" for each drive. 

 

If you have done this and you're still seeing performance issues, let me know.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#7 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 22 November 2016 - 11:29 AM

Yes I noticed very bad performance (not even close to the BW usage you see in my video) with the old keys (roughly around 20~60 mbps) so I updated on the 20th to the 770 build. Initial speeds was OK for a few hours (5-6), around 200~280 mbps with 100% I/O usage.

 

Yesterday speeds went back to the same as with the old api keys (40-80 mbps) and has been steady since then. I/O usage on the HDD is now at 30~50% so I'm sure it isn't that.

 

If Alex want's I'd let him remote controll the PC doing the upload, if he want's to debug.

PS: yes, I did re-authorize.



#8 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 22 November 2016 - 07:23 PM

Yes I noticed very bad performance (not even close to the BW usage you see in my video) with the old keys (roughly around 20~60 mbps) so I updated on the 20th to the 770 build. Initial speeds was OK for a few hours (5-6), around 200~280 mbps with 100% I/O usage.

 

Yesterday speeds went back to the same as with the old api keys (40-80 mbps) and has been steady since then. I/O usage on the HDD is now at 30~50% so I'm sure it isn't that.

 

If Alex want's I'd let him remote controll the PC doing the upload, if he want's to debug.

PS: yes, I did re-authorize.

 

 

Could you grab the logs from the system? 

 

And do you see a "turtle" icon next to the bandwidth usage? 

 

If so, then the slower speed may be because you're being throttled.  And in this case, reducing the number of threads may help.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#9 thnz

thnz

    Advanced Member

  • Members
  • PipPipPip
  • 139 posts

Posted 23 November 2016 - 08:01 AM

In the latest version (770), we've implemented an API pool, so that may help with performance. 

 

Hopefully Google doesn't take exception to that.



#10 jimbb

jimbb

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 November 2016 - 11:26 AM

I experience the same issue as well and on the latest version (770).

 

I can upload between 300-600 mbps on every fresh reboot, and after a while, it drops to 50-150 mbps and later dropped to 0-50 mbps. (On a 1Gbps line)

 

The weird thing is that on every fresh restart, I can go back to 300-600 mbps for a while until it drops again. From the look of it, it seems more like a CloudDrive issue rather than Google.

 

Tried changing the thread settings etc and nothing seems to help.

 

Already reported to support and waiting for a resolution :)



#11 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 23 November 2016 - 11:38 AM

I experience the same issue as well and on the latest version (770).

 

I can upload between 300-600 mbps on every fresh reboot, and after a while, it drops to 50-150 mbps and later dropped to 0-50 mbps. (On a 1Gbps line)

 

The weird thing is that on every fresh restart, I can go back to 300-600 mbps for a while until it drops again. From the look of it, it seems more like a CloudDrive issue rather than Google.

 

Tried changing the thread settings etc and nothing seems to help.

 

Already reported to support and waiting for a resolution :)

 

Exactly the same issue for me (just lower bandwidth  :D), I'll see if I can find some logs when I get home (OS logs or CloudDrive?)



#12 raidz

raidz

    Member

  • Members
  • PipPip
  • 12 posts

Posted 23 November 2016 - 09:39 PM

I am seeing the exact same issues as these two guys. I uploaded logs about a week ago so hopefully Alex can give this some time soon.



#13 jimbb

jimbb

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 November 2016 - 10:27 AM

Exactly the same issue for me (just lower bandwidth  :D), I'll see if I can find some logs when I get home (OS logs or CloudDrive?)

 

I am seeing the exact same issues as these two guys. I uploaded logs about a week ago so hopefully Alex can give this some time soon.

 

 

They came back to me this morning saying it might be Google drive who is limiting the API request frequent. However, I disagree as I have set the thread number to 1 on both upload and download and the speed drop is still significant comparing between fresh reboot (50-100mbps per thread) vs after 2-3 hours of run time (0-15 mbps per thread).

 

 They have advised me to tune up the chunk size as well which I created a new drive to test the theory, but still having the same issue over time :(

 

Will keep you guys as we go along. 

 



#14 raidz

raidz

    Member

  • Members
  • PipPip
  • 12 posts

Posted 24 November 2016 - 04:57 PM

My Chunk size is at 100mb, Minimum download at 10mb, Download threads at 12 and Upload Threads at 6. No matter how many threads I set (more or less) it is still slow as molasses (on a 1gbit/1gbit line).  Download speeds have decreased as well due to delay in thread response time. This started happening to me after .726 and has not been different since. 

 

I do believe this might has to do with Google limiting API request but also think it is something they changed in the software that is triggering it unless Google just made a big change to how they handle requests.



#15 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 27 November 2016 - 03:10 AM

Hopefully Google doesn't take exception to that.

 

Hopefully. :)

 

But we're not round-robining them API per request.  The re-authorization process picks one "at random", and the drive uses that key for the entire time it's mounted. 

 

If there is an issue with this behavior, we'll deal with at that time (or before, because yeah, I'm not a fan of it either)

 

 

 

 

They came back to me this morning saying it might be Google drive who is limiting the API request frequent. However, I disagree as I have set the thread number to 1 on both upload and download and the speed drop is still significant comparing between fresh reboot (50-100mbps per thread) vs after 2-3 hours of run time (0-15 mbps per thread).

 

 They have advised me to tune up the chunk size as well which I created a new drive to test the theory, but still having the same issue over time :(

 

Will keep you guys as we go along. 

 

Checking the text logs can verify that.  If you're seeing a lot of throttling requests, then that's what going on.

 

Otherwise, logging the traffic with FiddlerCap (http://www.telerik.c...dler/fiddlercap), usin the "decrypt https traffic" option may help us to identify the issue.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#16 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 28 November 2016 - 10:12 PM

Just thought I'd say this is still happening on .776 (initial speed is very good, after an hour or two it drops significantly). No indication in the text logs to as if this is rate or throttling of any kind.

 

This is on a brand new drive, created for the very purpose of testing this issue.

 

Settings: https://goo.gl/tp4ngZ



#17 steffenmand

steffenmand

    Advanced Member

  • Members
  • PipPipPip
  • 296 posts

Posted 29 November 2016 - 02:31 PM

It's the http response time that seems to increase over time - also happens here and nothing in the logs. A fresh reboot makes it all fine again! Takes some days for it to get really bad here though



#18 thnz

thnz

    Advanced Member

  • Members
  • PipPipPip
  • 139 posts

Posted 29 November 2016 - 08:47 PM

Perhaps the exponential backoff isn't resetting itself or something.



#19 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 30 November 2016 - 11:09 PM

Hmmm,  on an offchance, have you ran the network auto-tuning command? 

 

Specifically, try running the following from an administrative command line, and reboot.

 

netsh int tcp set global autotuninglevel=highlyrestricted


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#20 Joel Kåberg

Joel Kåberg

    Member

  • Members
  • PipPip
  • 15 posts

Posted 01 December 2016 - 11:00 PM

Hmmm,  on an offchance, have you ran the network auto-tuning command? 

 

Specifically, try running the following from an administrative command line, and reboot.

 

 

Yeah, I applied your suggested fix last night. Still same thing :-/






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users