Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Google Drive + PreFetch = Throttle?


  • Please log in to reply
11 replies to this topic

#1 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 18 April 2017 - 04:24 PM

Hey all,

I've run into an odd issue with CloudDrive and PreFetch. Essentially, with prefetch off, I get 300MB/s up and ~150MB/s down (when something is being pushed/pulled.) when I turn prefetch on, the upload downloads come to a screeching halt, uploads bob up and down from 0 to 15mb's and downloads generally stay under 1mb/s although they may occasionally go up to 8mb/s for a few seconds. Once I turn prefetch off and the time expires, the numbers spike back to their normal throughput. Here are my settings:

 

Google Drive attached

256TB AES Encrypted mount

10TB in use

20MB chunks

10MB/1 minute upload trigger

500/500 Internet with Gig-E link

Upload/download threads set to 5/5 (have tried 3/3 and 10/10 as well with same results.) Background I/O checked

Prefetch - Trigger 1MB, Forward 400MB, Time 1800 Seconds (tried 10/100/600, 1/1000/3600 with same results)

Upload Download limited to 250MB/250MB, although tried removing these limits for testing, same results

30GB expandable cache set on local SSD drive (tried increasing, reducing, changing to fixed or dynamic with same results)

Upload queues are at 0 prior to prefetch being enabled, but once enabled they start to grow due to the speed decrease

 

One thing I've noticed, I'f I'm understanding the summary screen correctly, is above the upload and download arrows it will show numbers like 3x or 5x. I'm assuming this is the number of threads being utilized. With prefetch off, this number never exceeds my max i/o settings for upload/download. However, with prefetch turned on, the download threads will go up, I've seen it go up to 12x for example. Does prefetch not respect the max i/o settings or am I missing something? Also, the prefetch fills slowly. Is the prefetch duration saying "if 1mb of same file is read, fetch next 400MB steadily/evenly spread over the next 1800 seconds" or is it saying "if 1mb of same file is read, fetch next 400MB as fast as possible keep for 1800 seconds."?

 

Also, what is the significance of the "Background I/O" checkbox next to upload threads? What does turning this option off and on do?

Thanks in advance,

-Ed



#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,287 posts
  • LocationSan Diego, CA, USA

Posted 19 April 2017 - 03:39 AM

Could you enable drive tracing, and reproduce this issue? 

http://wiki.covecube...e_Drive_Tracing

 

And could you record the web traffic? 

http://wiki.covecube...dDrive_Web_Logs


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 19 April 2017 - 05:23 AM

Done. Both submitted



#4 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 19 April 2017 - 05:35 AM

fwiw, heres a snip of the service logs as well. starts with prefetch on, I turn it off around 17:18

 

17:16:52.3: Warning: 0 : [ApiGoogleDrive:132] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:16:52.3: Warning: 0 : [ApiHttp:132] HTTP protocol exception (Code=Forbidden).
17:16:52.3: Warning: 0 : [IoManager:132] HTTP error (Forbidden) performing I/O operation on provider.
17:16:52.3: Warning: 0 : [IoManager:132] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:08.8: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:08.8: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden).
17:17:08.8: Warning: 0 : [IoManager:114] HTTP error (Forbidden) performing I/O operation on provider.
17:17:08.8: Warning: 0 : [IoManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiGoogleDrive:129] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiHttp:129] HTTP protocol exception (Code=Forbidden).
17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error on read when performing master partial write. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error when performing master partial write. The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiGoogleDrive:67] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:18.6: Warning: 0 : [ApiHttp:67] HTTP protocol exception (Code=Forbidden).
17:17:18.6: Warning: 0 : [IoManager:129] HTTP error (Forbidden) performing I/O operation on provider.
17:17:18.6: Warning: 0 : [IoManager:129] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error on read when performing master partial write. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error when performing master partial write. The download quota for this file has been exceeded.
17:17:18.7: Warning: 0 : [IoManager:67] HTTP error (Forbidden) performing I/O operation on provider.
17:17:18.7: Warning: 0 : [IoManager:67] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:21.5: Warning: 0 : [ApiGoogleDrive:108] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:21.5: Warning: 0 : [ApiHttp:108] HTTP protocol exception (Code=Forbidden).
17:17:21.5: Warning: 0 : [IoManager:108] HTTP error (Forbidden) performing I/O operation on provider.
17:17:21.5: Warning: 0 : [IoManager:108] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:17:38.5: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded.
17:17:38.5: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden).
17:17:38.5: Warning: 0 : [IoManager:114] HTTP error (Forbidden) performing I/O operation on provider.
17:17:38.5: Warning: 0 : [IoManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded.
17:18:31.1: Information: 0 : [IoManager:3] All writes complete.
17:19:05.6: Information: 0 : [IoManager:3] All writes complete.


#5 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,287 posts
  • LocationSan Diego, CA, USA

Posted 20 April 2017 - 07:35 PM

edrock200, 

 

I believe that you've posted this in a ticket, at well, and I've responded there, already. 

 

That said, try disabling the prefetcher, and see if that helps. 

 

Otherwise, do this: 

http://wiki.covecube...e_Drive_Tracing


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#6 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 21 April 2017 - 03:05 AM

Chris,

Not sure what you mean by posted in a ticket, I posted it via the dropbox link you provided. How do I see your response in the ticket? I have kept prefetch off, as the drive is unusable when its on. It works ok with prefetch off but I still hit a lot of buffering, so I'm trying to get prefetch working to alleviate this.



#7 andy0123456

andy0123456

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 21 April 2017 - 07:49 PM

With the speeds you have, I think what you're looking for is increasing the prefetcher chunk size. This is part of the drive attach process and is labelled "minimum download size".

 

By default the prefetcher operates in 1mb chunks, I assume it sacrifices throughput for response time, but it gets quickly throttled by google, for example my drive is unused most of today and google starts throttling me after ~10 seconds of prefetching at 1mb chunks.



#8 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 22 April 2017 - 04:52 AM

Thank you. Where is that set? I see prefetch trigger but I though that's the threshold that initiates prefetching

#9 andy0123456

andy0123456

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 22 April 2017 - 06:43 AM

Setting is only changeable when attaching or creating drives, it's "minimum download size". It can't be set higher then cache chunk size.



#10 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted 22 April 2017 - 02:02 PM

Sweet. Thanks. I'll try that

#11 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,287 posts
  • LocationSan Diego, CA, USA

Posted Yesterday, 12:11 AM

  • Prefetch Trigger: This is the amount of data that needs to be read sequentially in order to trigger the prefetcher.
  • Prefetch Forward: Once prefetching is triggered, this is the amount of data that will be pre-downloaded ahead of the current read request.
  • Prefetch Time Window: This controls how quickly the sequential read requests have to occur in order to trigger the prefetcher.

 

So the trigger is how much it has to read sequentially, prefetch time window is the window of time that this has to occur in or it gets ignored, and prefetch forward is how much data it goes out and grabs., 

 

 

Setting the "Minimum download size" changes how many API calls are made, as it grabs larger chunks of data for the partial reads, but increases latency (it takes longer to download 10MB than 1MB), so it can introduce some "lagginess" to the drive (though the pinning may/should help with this, at least a bit). 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#12 Edrock200

Edrock200

    Member

  • Members
  • PipPip
  • 10 posts

Posted Today, 03:59 AM

Thanks for that explanation. If I'm understanding it correctly, I think many who have been using clouddrive for plex do not understand the time window setting. There are numerous "tutorial" articles that suggest setting the time window to 1mb/400mb/3600seconds, which would cause massive, neverending prefetching. The tutorial writers seem to think the time window is how long the prefetch data will be stored in cache for.

 

Did you get a chance to look at the logs I uploaded to see why prefetching causes things to throttle? Right now, if I turn prefetching on with any settings, even the defaults, my drive will disconnect within 5 to 10 minutes with an error that it cannot read from the provider.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users