Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Google Drive + PreFetch = Throttle?


Best Answer Edrock200 , 29 April 2017 - 05:07 PM

In case anyone else comes across this thread who is also using Plex, don't let Plex scan your drive until your initial upload is done. I have mine set not to generate thumbnails or due deep media analysis, but it still makes a lot of calls to the drive on initial scan which will potentially cause throttling. Lessons learned on my end:

Create drive
Turn prefetch off
Turn off excessive indexers and Google sync clients if your drive is using Google storage*
Preload drive with content
Turn on media scanners and let initial scan complete
Turn on prefetch

If you try to do it all in parallel you will most likely get throttled and it will probably take longer than doing it in the order above. Since Google drive client doesn't offer any detailed stats, I didn't realize it had essentially slowed to zero.

*If you are using Google drive client to preload your clouddrive, throttle drive client up/down to 99/99mbs (max it will allow) and cloud drive to 250/250 up down with 5/5 io threads up/down and no prefetch.

Should you find that you need to move your drive to another pc and still want drive client linked to cloudrive for syncing, do this:

Old PC:
Close Google drive client
Backup folder %localappdata%\google\drive
Detach cloud drive

New pc:
Install Google drive client and authenticate to account. Accept defaults but immediately close client after logon
Mount clouddrive to SAME drive letter as old PC
Restore %localappdata%\google\drive from backup to new pc
Relaunch Google drive client

This will restore the drive client file hashes and prevent your new pc from needing to download and hash every single file again. Alternatively you can open regedit and go to hkcu/software/Google/drive and point the appdata directory to your cloud drive and restore to that directory, so the hash stays with the drive. Since it's frequently used it should stay in cache when mounted.
Ed
*you can turn prefetch back on after initial load of files onto cloud drive
*I did not mean to mark my own reply as best answer. I was just trying to mark the entire thread as solved. :) Go to the full post


  • Please log in to reply
14 replies to this topic

#1 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 18 April 2017 - 04:24 PM

Hey all,

I've run into an odd issue with CloudDrive and PreFetch. Essentially, with prefetch off, I get 300MB/s up and ~150MB/s down (when something is being pushed/pulled.) when I turn prefetch on, the upload downloads come to a screeching halt, uploads bob up and down from 0 to 15mb's and downloads generally stay under 1mb/s although they may occasionally go up to 8mb/s for a few seconds. Once I turn prefetch off and the time expires, the numbers spike back to their normal throughput. Here are my settings:

 

Google Drive attached

256TB AES Encrypted mount

10TB in use

20MB chunks

10MB/1 minute upload trigger

500/500 Internet with Gig-E link

Upload/download threads set to 5/5 (have tried 3/3 and 10/10 as well with same results.) Background I/O checked

Prefetch - Trigger 1MB, Forward 400MB, Time 1800 Seconds (tried 10/100/600, 1/1000/3600 with same results)

Upload Download limited to 250MB/250MB, although tried removing these limits for testing, same results

30GB expandable cache set on local SSD drive (tried increasing, reducing, changing to fixed or dynamic with same results)

Upload queues are at 0 prior to prefetch being enabled, but once enabled they start to grow due to the speed decrease

 

One thing I've noticed, I'f I'm understanding the summary screen correctly, is above the upload and download arrows it will show numbers like 3x or 5x. I'm assuming this is the number of threads being utilized. With prefetch off, this number never exceeds my max i/o settings for upload/download. However, with prefetch turned on, the download threads will go up, I've seen it go up to 12x for example. Does prefetch not respect the max i/o settings or am I missing something? Also, the prefetch fills slowly. Is the prefetch duration saying "if 1mb of same file is read, fetch next 400MB steadily/evenly spread over the next 1800 seconds" or is it saying "if 1mb of same file is read, fetch next 400MB as fast as possible keep for 1800 seconds."?

 

Also, what is the significance of the "Background I/O" checkbox next to upload threads? What does turning this option off and on do?

Thanks in advance,

-Ed


  • Antoineki likes this

#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 19 April 2017 - 03:39 AM

Could you enable drive tracing, and reproduce this issue? 

http://wiki.covecube...e_Drive_Tracing

 

And could you record the web traffic? 

http://wiki.covecube...dDrive_Web_Logs


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 19 April 2017 - 05:23 AM

Done. Both submitted



#4 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 19 April 2017 - 05:35 AM

fwiw, heres a snip of the service logs as well. starts with prefetch on, I turn it off around 17:18

 

17:16:52.3: Warning: 0 : [ApiGoogleDrive:132] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:16:52.3: Warning: 0 : [ApiHttp:132] HTTP protocol exception (Code=Forbidden).
17:16:52.3: Warning: 0 : [IoManager:132] HTTP error (Forbidden) performing I/O operation on provider.
17:16:52.3: Warning: 0 : [IoManager:132] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:08.8: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:08.8: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden).
17:17:08.8: Warning: 0 : [IoManager:114] HTTP error (Forbidden) performing I/O operation on provider.
17:17:08.8: Warning: 0 : [IoManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiGoogleDrive:129] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiHttp:129] HTTP protocol exception (Code=Forbidden).
17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error on read when performing master partial write. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error when performing master partial write. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiGoogleDrive:67] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiHttp:67] HTTP protocol exception (Code=Forbidden).
17:17:18.6: Warning: 0 : [IoManager:129] HTTP error (Forbidden) performing I/O operation on provider.
17:17:18.6: Warning: 0 : [IoManager:129] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error on read when performing master partial write. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error when performing master partial write. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [IoManager:67] HTTP error (Forbidden) performing I/O operation on provider.
17:17:18.7: Warning: 0 : [IoManager:67] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:21.5: Warning: 0 : [ApiGoogleDrive:108] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:21.5: Warning: 0 : [ApiHttp:108] HTTP protocol exception (Code=Forbidden).
17:17:21.5: Warning: 0 : [IoManager:108] HTTP error (Forbidden) performing I/O operation on provider.
17:17:21.5: Warning: 0 : [IoManager:108] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:38.5: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:38.5: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden).
17:17:38.5: Warning: 0 : [IoManager:114] HTTP error (Forbidden) performing I/O operation on provider.
17:17:38.5: Warning: 0 : [IoManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:18:31.1: Information: 0 : [IoManager:3] All writes complete.
17:19:05.6: Information: 0 : [IoManager:3] All writes complete.


#5 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 20 April 2017 - 07:35 PM

edrock200, 

 

I believe that you've posted this in a ticket, at well, and I've responded there, already. 

 

That said, try disabling the prefetcher, and see if that helps. 

 

Otherwise, do this: 

http://wiki.covecube...e_Drive_Tracing


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#6 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 21 April 2017 - 03:05 AM

Chris,

Not sure what you mean by posted in a ticket, I posted it via the dropbox link you provided. How do I see your response in the ticket? I have kept prefetch off, as the drive is unusable when its on. It works ok with prefetch off but I still hit a lot of buffering, so I'm trying to get prefetch working to alleviate this.



#7 andy0123456

andy0123456

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 21 April 2017 - 07:49 PM

With the speeds you have, I think what you're looking for is increasing the prefetcher chunk size. This is part of the drive attach process and is labelled "minimum download size".

 

By default the prefetcher operates in 1mb chunks, I assume it sacrifices throughput for response time, but it gets quickly throttled by google, for example my drive is unused most of today and google starts throttling me after ~10 seconds of prefetching at 1mb chunks.



#8 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 22 April 2017 - 04:52 AM

Thank you. Where is that set? I see prefetch trigger but I though that's the threshold that initiates prefetching

#9 andy0123456

andy0123456

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 22 April 2017 - 06:43 AM

Setting is only changeable when attaching or creating drives, it's "minimum download size". It can't be set higher then cache chunk size.


  • Edrock200 likes this

#10 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 22 April 2017 - 02:02 PM

Sweet. Thanks. I'll try that

#11 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 27 April 2017 - 12:11 AM

  • Prefetch Trigger: This is the amount of data that needs to be read sequentially in order to trigger the prefetcher.
  • Prefetch Forward: Once prefetching is triggered, this is the amount of data that will be pre-downloaded ahead of the current read request.
  • Prefetch Time Window: This controls how quickly the sequential read requests have to occur in order to trigger the prefetcher.

 

So the trigger is how much it has to read sequentially, prefetch time window is the window of time that this has to occur in or it gets ignored, and prefetch forward is how much data it goes out and grabs., 

 

 

Setting the "Minimum download size" changes how many API calls are made, as it grabs larger chunks of data for the partial reads, but increases latency (it takes longer to download 10MB than 1MB), so it can introduce some "lagginess" to the drive (though the pinning may/should help with this, at least a bit). 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#12 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 28 April 2017 - 03:59 AM

Thanks for that explanation. If I'm understanding it correctly, I think many who have been using clouddrive for plex do not understand the time window setting. There are numerous "tutorial" articles that suggest setting the time window to 1mb/400mb/3600seconds, which would cause massive, neverending prefetching. The tutorial writers seem to think the time window is how long the prefetch data will be stored in cache for.

 

Did you get a chance to look at the logs I uploaded to see why prefetching causes things to throttle? Right now, if I turn prefetching on with any settings, even the defaults, my drive will disconnect within 5 to 10 minutes with an error that it cannot read from the provider.



#13 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,340 posts
  • LocationSan Diego, CA, USA

Posted 29 April 2017 - 12:08 AM

Yeah, there are, and the 3600 window is a misunderstanding of what the value does. 

 

That is in part my fault, as well.  

 

You generally want to leave the time window at a smaller value, or just about every request may get prefetched. 

 

 

As for the logs, I'm mostly seeing these: "Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded". 

 

This was a known issue on older versions/drives, because we were querying the files too much.  That said, if you're using a recent version, this really shouldn't be happening. But we've seen a number of reports of this recently. I've already flagged the issue for Alex, so he an take a look into this. 

 

In the meanwhile, detaching and reattaching the drive may fix the issue. Specifically, when re-attaching the drive, open the "Advanced Settings" section, and set the "Minimum Download Size" to 5MB or higher, and then attach it. 

 

This will help avoid this error. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#14 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 29 April 2017 - 07:51 AM

Thanks. I think I figured it out my issue. First, I reattached the drive with 20mb min download limit set. Initially this still did not help. Next, and I could be wrong on this, but prefetch does not appear to respect the max thread setting. Ie if I set it to 5 max threads, and had prefetch set to 100mb in 20mb chunks, any prefetch calls would cause a connection spike to 10x.

I also found that my Google drive client, which was fully synced, had reinitiated about 1tb of download, due to the fact that my other pc with Google drive had a file scan operation "touch" all the files thereby modifying the file date, those this PC was trying to download these "new" files.

It seems that Google not only throttles by api id, but also by ip. It seems to start throttling everything when sustained traffic to/from one source stays steady over 600mbs. I noticed when I paused Google drive, clouddrive jumped back up to full speed with prefetch working great. Then when i unpaused Google drive, it took nearly an hour to download a 1gb file, as though Google drive was now throttling. And this wasn't a bandwidth constraint on my end, I have 1gig link to the internet with speed tests running about 900/880mbs, and with both running I was at about 250 up and 400 down.

To resolve this my final settings are as follows, but please note my drive content is primarily large video media:
Min download 20mb
Download chunk cache 100mb
Download/upload throttle cloud drive 250mbs
Download/upload threads 5/5
Prefetch 10mb/100mb/120s

Google drive download throttle 99mbs (max throttle value allowed)

From what I can tell, if Google sees the same IP pushing and/or pulling a sustained rate greater than 600mbs it gets much more strict on throttling and threads. The above settings seem stable now and I even tested 5 simultaneous hd streams without issue. But launching the Google music sync client, which literally just opened a few file comparison threads, caused the numbers to drop almost instantly. Closed it, and they went right back up. Throttled it to 10mbs and all was happy again.

I do believe the prefetch threading issue should be given a closer look though. I specifically set mine to 100mbs because with 20mb download chunks, each prefetch will cause a Max of 5 new threads. Couple that with the existing Max of 5 dl threads and I'm still in a reasonable thread range of 10. When I tested 200mbs, it still worked, but the first 100 prefetched almost instantly, then downloads started to slow and the last 100 came in throttled. Tested 300 and it made it to 200, then slowed to the single digits, got to 360, slowed to kilobit range, then my drive disconnected.

This is all very unscientific and just my experience.

The other tweak I did was turn off windows built in drive indexing for the cloud drive to minimize "idle thrashing."

The good news for me is my initial copy to the cloud drive is 95% complete, and should finish this weekend so all this bandwidth excess and throttling issues should subside and future writes will only be a few gb/day.

Thanks all for the info and advice. :)

#15 Edrock200

Edrock200

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 29 April 2017 - 05:07 PM   Best Answer

In case anyone else comes across this thread who is also using Plex, don't let Plex scan your drive until your initial upload is done. I have mine set not to generate thumbnails or due deep media analysis, but it still makes a lot of calls to the drive on initial scan which will potentially cause throttling. Lessons learned on my end:

Create drive
Turn prefetch off
Turn off excessive indexers and Google sync clients if your drive is using Google storage*
Preload drive with content
Turn on media scanners and let initial scan complete
Turn on prefetch

If you try to do it all in parallel you will most likely get throttled and it will probably take longer than doing it in the order above. Since Google drive client doesn't offer any detailed stats, I didn't realize it had essentially slowed to zero.

*If you are using Google drive client to preload your clouddrive, throttle drive client up/down to 99/99mbs (max it will allow) and cloud drive to 250/250 up down with 5/5 io threads up/down and no prefetch.

Should you find that you need to move your drive to another pc and still want drive client linked to cloudrive for syncing, do this:

Old PC:
Close Google drive client
Backup folder %localappdata%\google\drive
Detach cloud drive

New pc:
Install Google drive client and authenticate to account. Accept defaults but immediately close client after logon
Mount clouddrive to SAME drive letter as old PC
Restore %localappdata%\google\drive from backup to new pc
Relaunch Google drive client

This will restore the drive client file hashes and prevent your new pc from needing to download and hash every single file again. Alternatively you can open regedit and go to hkcu/software/Google/drive and point the appdata directory to your cloud drive and restore to that directory, so the hash stays with the drive. Since it's frequently used it should stay in cache when mounted.
Ed
*you can turn prefetch back on after initial load of files onto cloud drive
*I did not mean to mark my own reply as best answer. I was just trying to mark the entire thread as solved. :)




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users