Jump to content
  • 0

Clouddrive, Gsuite unlimited, and Plex


Middge

Question

Hey all, long time user of Clouddrive here.

A long long time ago (a year or two ago?) back when Clouddrive was in beta, I bought the product and began using it. It met almost all of my expectations and exceeded others. I use clouddrive to store media like most others here and I've stored almost 80TB thus far.

Here is my problem,

I created the drive using a very early beta, and I think my disk is actually suffering for this. It seems to get worse as I store more data. I want to start over, but this time I want to make sure I do it right. Is there any document or guide somewhere that can tell me the best practices for my use-case? That is, streaming high quality HD media over 10-20 concurrent connections with 1gbps symmetrical bandwidth. What chunk size should I use? What about cache size and prefetching?

My current setup is as follows;

10MB chunk size (i was wondering if I should increase to 20MB?)

1MB prefetch trigger

20MB prefetch size (should this match chunk size?)

240 seconds prefetch time window

20 upload threads

20 download threads

Background I/O: DISABLED

50GB Cache on SSD (Should I increase this? I can probably afford up to 500GB SSD cache)

Drive was formatted as NTFS (Should I use REFS? i am leary of this as refs has higher overhead and little recourse in the event of failure)

Any help that you guys could provide would be greatly appreciated. Thanks in advance for your time.

Link to comment
Share on other sites

5 answers to this question

Recommended Posts

  • 1
10 hours ago, Christopher (Drashna) said:

First, what provider is this for?  Each has very different "profiles". 

More threads isn't always better.  Too many can trigger DoS protection, and can run into API limits, as well. 

It says Gsuite in the title. I'm assuming that means Google Drive. Correct me if all of the following is wrong, Middge. 

 

15 hours ago, Middge said:

Hey all, long time user of Clouddrive here.
 

Hey Middge,

I use a CloudDrive for Plex with Google Drive myself. I can frequently do 5-6 remux quality streams at once. I haven't noticed a drop in capacity aside from Google's relatively new upload limits. 

Quote

10MB chunk size (i was wondering if I should increase to 20MB?)

Yes. But this is going to require that you remake the drive. My server also has a fast pipe, and I've also raised the minimum download to 20MB as well. I really haven't noticed any slowdown in responsiveness because the connection is so fast, and it keeps the overall throughput up.

Quote

1MB prefetch trigger

This is fine. You can play with it a bit. Some people like higher numbers like 5 or 10 MB triggers, but I've tried those and I keep going back to 1 MB as well, because I've just found it to be the most consistent, performance-wise, and I really want it to grab a chunk of *all* streaming media immediately.

Quote

20MB prefetch size (should this match chunk size?)

This is way too low, for a lot of media. I would raise this to somewhere between 150-300MB. Think of the prefetch as a rolling buffer. It will continue to fill the prefetch as the data in the prefetch is used. The higher the number, then, the more tolerant your stream will be to periodic network hiccups. The only real danger is that if you make it too large (and I'm talking like more than 500MB here) it will basically always be prefetching, and you'll congest your connection if you hit like 4 or 5 streams. 

Quote

240 seconds prefetch time window

I would drop this to 30, no matter whether you want to go with a 1MB, 5MB, or 10MB trigger. 240 seconds almost makes the trigger amount pointless anyway--you're going to hit all of those benchmarks in 4 minutes if you're streaming most modern media files. A 30 second window should be fine.

Quote

20 upload threads

20 download threads

WAAAAY too many. You're almost certainly throttling yourself with this. Particularly with the smaller than maximum chunk size, since it already has to make more requests than if you were using 20MB chunks. I use three clouddrives in a pool (a legacy arrangement from before I understood things better, don't do it. Just expand a single clouddrive with additional volumes), and I keep them all at 5 upload and 5 download threads. Even if I had a single drive, I'd probably not exceed 5 upload, 10 download. 20 and 20 is *way* too high and entirely unnecessary with a 1gbps connection. 

Quote

Background I/O: DISABLED

50GB Cache on SSD (Should I increase this? I can probably afford up to 500GB SSD cache)

Drive was formatted as NTFS (Should I use REFS? i am leary of this as refs has higher overhead and little recourse in the event of failure)

These are all fine. If you can afford a larger cache, bigger is *always* better. But it isn't necessary. The server I use only has 3x 140GB SSDs, so my caches are even smaller than yours and still function great. The fast connection goes a long way toward making up for a small cache size...but if I could have a 500GB cache I definitely would. 

Link to comment
Share on other sites

  • 1
1 hour ago, srcrist said:

It says Gsuite in the title. I'm assuming that means Google Drive. Correct me if all of the following is wrong, Middge. 

Ugh, blind, apparently!  I was only looking through the post text.... /sigh (only my fault for being an idiot)

 

In this case: 

16 hours ago, Middge said:

20 upload threads

20 download threads

No. This will cause you to get throttled into oblivion.  2-5 threads EACH. At most.  At least, at first.  Then slowly increase the number until you start getting throttling notifications.  Otherwise, 20 WILL get you throttled IMMEDIATELY. 

16 hours ago, Middge said:

10MB chunk size (i was wondering if I should increase to 20MB?)

Honestly, use the largest that you can, here.

But you should also set the minimum download size to 5MB or larger. And the larger this is, the higher the latency may be. 

16 hours ago, Middge said:

1MB prefetch trigger

20MB prefetch size (should this match chunk size?)

240 seconds prefetch time window

I'd say 5/100MB/30, actually. 

A higher window is bad. This is the timeframe in which prefetching can be triggered. And 1MB? that means any sequential 1MB in 240 seconds will trigger it, and download 20MB.  That may be actually far from ideal. 

5/100/30 will look for 5MB sequential reads, that happen in a 30 second window, and then prefetch the next 100MBs  This may be more in line with what you want. 

16 hours ago, Middge said:

Background I/O: DISABLED

This may adversely affect system performance, overall.  
The background IO option means that other IO (eg local) happens at a higher priority, so as not to impact system performance. 

It shouldn't be "bad", but it's an optimization to make sure the system feels properly responsive. 

16 hours ago, Middge said:

50GB Cache on SSD (Should I increase this? I can probably afford up to 500GB SSD cache)

SSDs are great for this. Just remember that they're going to get thrashed pretty bad.  Using a Samsung with their Magician software to allocate more "buffer space" may be a good idea (or any drive that allows this).

But the larger, the better, honestly. 

Also, if you use the default (expandable), it will use more than the 50GB, when writing to the CloudDrive disk. 

But this will help to negate any impact that turning off background I/O may have, actually. 

16 hours ago, Middge said:

Drive was formatted as NTFS (Should I use REFS? i am leary of this as refs has higher overhead and little recourse in the event of failure)

NTFS should be fine.  We enable "checksum verification" by default.  that means that every chunk uploaded and downloaded has a checksum that is written and checked.  ReFS would be redundant, at that point, actually (as it can't self heal if it's not in a Storage Spaces array). 

1 hour ago, srcrist said:

I haven't noticed a drop in capacity aside from Google's relatively new upload limits. 

Yup, that .... And their help page isn't helpful. It shows the limit for some accounts (it looks like it only show it for business/gsuite accounts) 

But in either case, you should throttle the upload to around 75mbps, as this will come close to the 750GB daily limit, but not exceed it. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...