Warning from GDrive (Plex)

[Chupa]

Hi guys, I have a problem with CloudDrive I hope that those who are more experienced than me can help me. Thanks

I use CloudDrive with Plex and I have about 10 to 30 simultaneous streams, I have a 1gbps / 1gbps connection and the operating system and metadata both run on SSD and I never have problems using Plex even when I reach the maximum peak of 30 streams, different is when I try to upload, even having only 1 stream, CloudDrive gives me this error. After this error I find all the files loaded correctly on the drive, but I don't understand what error it is and if it can generate other problems.

 

 

My configuration is this: I have 30TB drives in NTFS, chunk size is 20MB and the cache on SSD and is set 50GB +

Download threads: 9
Upload threads: 9
Backgruond I/O: YES
Upload Threshold: 1 MB or 5 minutes
Minimum download size: 20MB

Prefetch trigger: 20MB
Prefetch forward: 180 MB
Prefetch time windows: 30 seconds

 

Probably something wrong in the configurations but I don't understand why if I don't upload I never get these warnings. I thank in advance those who can help me

[srcrist]

That's just a warning. You thread count is a bit too high, and you're probably getting throttled. Google only allows around 15 simultaneous threads at a time. Try dropping your upload threads to 5 and keeping your download threads where they are. That warning will probably go away.

Ultimately, though, even temporary network hiccups can occasionally cause those warnings. So it might also be nothing. It's only something to worry about if it happens regularly and frequently. 

[Chupa]

ok I will try to change the number of threads in upload, I will keep you updated because these warnings occur every time I send something to the drive. Thank you very much

[Christopher (Drashna)]

Thread count is fine. We really haven't seen issues with 10.  

However, the settings you have set WILL cause bottlenecking and issues. 

Download threads: 10
Upload threads: 10
Minimum download size: 20MB

Prefetch trigger: 5MB
Prefetch forward: 150 MB
Prefetch time windows: 30 seconds

 

The Prefetch forward should be roughly 75% of download threads x minimum download size.   If you can set a higher minimum size, then you can increase the forward.  

 

[srcrist]

3 hours ago, Christopher (Drashna) said:

Thread count is fine. We really haven't seen issues with 10.  

However, the settings you have set WILL cause bottlenecking and issues. 

Download threads: 10
Upload threads: 10
Minimum download size: 20MB

Prefetch trigger: 5MB
Prefetch forward: 150 MB
Prefetch time windows: 30 seconds

 

The Prefetch forward should be roughly 75% of download threads x minimum download size.   If you can set a higher minimum size, then you can increase the forward.  

 

Out of curiosity, does Google set different limits for the upload and download threads in the API? I've always assumed that since I see throttling around 12-15 threads in one direction, that the total number of threads in both directions needed to be less than that. Are you saying it should be fine with 10 in each direction even though 20 in one direction would get throttled?

[MandalorePatriot]

18 hours ago, srcrist said:

Out of curiosity, does Google set different limits for the upload and download threads in the API? I've always assumed that since I see throttling around 12-15 threads in one direction, that the total number of threads in both directions needed to be less than that. Are you saying it should be fine with 10 in each direction even though 20 in one direction would get throttled?

What is even the gain of using so many threads? More connections, sure, but doesn't Google throttle bandwidth after a certain amount? And it also depends on your upload speed, I'm capped by IPS @ 40Mbps roughly, so it seems only 3 upload threads is plenty.

[srcrist]

1 hour ago, MandalorePatriot said:

What is even the gain of using so many threads? More connections, sure, but doesn't Google throttle bandwidth after a certain amount? And it also depends on your upload speed, I'm capped by IPS @ 40Mbps roughly, so it seems only 3 upload threads is plenty.

To my knowledge, Google does not throttle bandwidth at all, no. But they do have the upload limit of 750GB/day, which means that a large number of upload threads is relatively pointless if you're constantly uploading large amounts of data. It's pretty easy to hit 75mbps or so with only 2 or 3 upload threads, and anything more than that will exceed Google's upload limit anyway. If you *know* that you're uploading less than 750GB that day anyway, though, you could theoretically get several hundred mbps performance out of 10 threads. So it's sort of situational.

Many of us do use servers with 1gbps synchronous pipes, in any case, so there is a performance benefit to more threads...at least in the short term. 

But, ultimately, I'm mostly just interested in understanding the technical details from Christopher so that I can experiment and tweak. I just feel like I have a fundamental misunderstanding of how the API limits work. 

[Christopher (Drashna)]

23 hours ago, srcrist said:

Out of curiosity, does Google set different limits for the upload and download threads in the API? I've always assumed that since I see throttling around 12-15 threads in one direction, that the total number of threads in both directions needed to be less than that. Are you saying it should be fine with 10 in each direction even though 20 in one direction would get throttled?

I'm not sure?  But the number of threads is set by our program. Mostly, it's just the number of open/active connections. 

Also, given how uploading is handled, the upload threshold may help prevent this from being an issue. But you can reduce the upload threads, if you want. 

5 hours ago, MandalorePatriot said:

What is even the gain of using so many threads? More connections, sure, but doesn't Google throttle bandwidth after a certain amount? And it also depends on your upload speed, I'm capped by IPS @ 40Mbps roughly, so it seems only 3 upload threads is plenty.

Parallel connections.  For stuff like prefetching, it makes a different.  Or if you have a lot of random access on the drives...

But otherwise, they do have the daily upload limit, and they will throttle for other reasons (eg, DOS/DDoS protection)

[Bowsa]

Is this over-kill then? I have these settings to ensure I upload my files as fast as possible (1000/1000) whilst also being able to Stream 4K and high-bitrate content with no hiccups.

[Christopher (Drashna)]

Well, I'd recommend increasing the minimum download size to the highest value that you can set, decrease the number of threads, and the trigger size. 

 

And as I said above: 

On 4/2/2019 at 3:20 PM, Christopher (Drashna) said:

The Prefetch forward should be roughly 75% of download threads x minimum download size.   If you can set a higher minimum size, then you can increase the forward.