Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

GSuite, Cloud Drive, Plex, Gigabit, Prefetch Settings


  • Please log in to reply
10 replies to this topic

#1 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 12 October 2017 - 10:08 PM

Hi all,

 

I have a home 1 Gb/s up/down connection and a very beefy Plex server. I use Cloud Drive with GSuite and a 500 GB SSD cache drive. 90% of my content is high bitrate 1080p video. I was wondering what the ideal I/O performance settings were.

 

I get pretty good performance as it is with these settings, but some things are a bit unclear and I'd like to optimize things:

 

Download threads: 10

Minimum download size: 100 MB

Prefetch trigger: 10 MB

Prefetch forward: 3000 MB

Prefetch time window: 30 seconds

 

It seems the prefetcher isn't too intelligent on the file level. That is, if I'm accessing a 1000 MB file, it still prefetches 3000 MB of data. But from where? Just the next contiguous 2000 MB of blocks? Ideally I'd want to set the prefetch forward to something like 15000 MB which would be the size of my largest media files, but then I think it'd prefetch 15 GB of data on every prefetch, regardless if it's relevant.

 

How does the minimum download size work with the prefetcher? It's my understanding that essentially if your cloud block size < minimum download size, it says to always download however many blocks it needs to fill up the minimum download size. How does that work with the prefetcher?

 

I'm still unclear how the prefetch trigger and the time window relate. In another thread I think it essentially says if 10 MB of data is accessed for 30 seconds, prefetch 3000 MB? I'm not exactly sure how that would work with a gigabit connection. As 10 MB would never take 30 seconds to download.

 

All that aside, if I have a gigabit connection, does the prefetcher actually help? It seems that sometimes there are download activities going on alongside prefetch activities and I wonder if they cause issues. Any other optimization tips?

 

Thanks!

 



#2 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 13 October 2017 - 12:05 AM

Ok, did some experimenting!

 

So, the prefetcher from what I understand now is essentially another, higher priority, "cache", and the window is how long the prefetcher keeps that data around. I've observed DRASTICALLY better performance by increasing the trigger to 100 MB and the window to around 30 minutes.

 

I also increased Transcoder default throttle buffer in Plex to 10 minutes so that it will go ahead and keep requesting additional data.

 

Another question: does data ever move from the prefetcher into the normal cache or is it dropped after your set window no matter what?



#3 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 13 October 2017 - 03:02 PM

The window is actually a threshold that triggers the prefetch if the trigger amount of data is requested within that time. For plex, I would leave your trigger at 1MB and the window at something like 30 secs. 150MB forward prefetch should be sufficient for even the highest bitrate video files. 

 

You also should definitely lower that Plex transcoder throttle time. That's 10 minutes of CPU time, not 10 minutes of normal old clock time. That's *waaaaaay* too much. Leave it at like 60 secs to 120 secs. 


  • Christopher (Drashna) likes this

#4 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 13 October 2017 - 06:33 PM

The window is actually a threshold that triggers the prefetch if the trigger amount of data is requested within that time. For plex, I would leave your trigger at 1MB and the window at something like 30 secs. 150MB forward prefetch should be sufficient for even the highest bitrate video files. 

 

You also should definitely lower that Plex transcoder throttle time. That's 10 minutes of CPU time, not 10 minutes of normal old clock time. That's *waaaaaay* too much. Leave it at like 60 secs to 120 secs. 

The window does not appear to function like that. You can change the window and watch the prefetched data get held in the UI for the time you set. Try setting the window to 30 minutes and set a timer, it'll hold that prefetched data for 30 minutes and then drop it. How would 1 MB in 30 seconds work? If it did work how you are saying, how do you access 1 MB in 30 seconds? It would take like a couple seconds at MOST to access that much data so it would never trigger that "window". The other issue with a low trigger size is that if Plex or any other program is analyzing file metadata, it triggers the prefetcher and clogs up it up even if it only needs to read parts of an entire file.

 

Do you have any proof that the Transcoder default throttle buffer is CPU time? "Amount in seconds to buffer before throttling back the transcoder speed." is what the Plex site says and I can't find any other sources other than suggested amounts on the Rclone forums of 10 minutes. Even if it is CPU time, it tells Plex to keep trancoding additional data and the Plex client will buffer and cache it.

 

In any case, I am certainly getting better performance with these settings. I can see spikes of data streaming / trancoding when the large prefetch gets downloaded and then processed by Plex and sent to the client. I watched 4x 1080p streams get handled flawlessly. That being said, I do have gigabit and 20x 2 GHz cores on the server.



#5 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,394 posts
  • LocationSan Diego, CA, USA

Posted 13 October 2017 - 07:03 PM

  • Prefetch Trigger
    This is the amount of data that needs to be read sequentially in order to trigger the prefetcher.
  • Prefetch Forward
    Once prefetching is triggered, this is the amount of data that will be pre-downloaded ahead of the current read request.
  • Prefetch Time Window
    This controls how quickly the sequential read requests have to occur in order to trigger the prefetcher

So, you generally want smaller values for this.  

 

10MB/100MB/30s is a good value, if you want a lot to be downloaded.

 

The prefetch trigger has to happen within the prefetch time window.
 
For example, with the prefetch trigger set to 10 MB and the prefetch time window set to 30 seconds, if an application reads 10MB of a file sequentially, but it does so very slowly and takes longer than 30 seconds to do it, that won't trigger the prefetcher.
 

 

And this is a "rolling" prefetcher, Meaning that as long as data keeps getting read, it will keep going out and prefetching it. 

 

 

 

And yes, the prefetcher (and generally, all of CloudDrive) is file system agnostic.  It doesn't look at the files on the drive, nor the access of them. It looks at how the blocks of data are being accessed. 

 

 

And yes, the prefetcher is aware of the minimum download size, and will actually grab blocks in that chunk, with extra data (eg, if it's "too much") being retained, as well. 


  • sysops likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#6 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 13 October 2017 - 07:20 PM

 

 

The window does not appear to function like that. You can change the window and watch the prefetched data get held in the UI for the time you set. Try setting the window to 30 minutes and set a timer, it'll hold that prefetched data for 30 minutes and then drop it. How would 1 MB in 30 seconds work? If it did work how you are saying, how do you access 1 MB in 30 seconds? It would take like a couple seconds at MOST to access that much data so it would never trigger that "window". The other issue with a low trigger size is that if Plex or any other program is analyzing file metadata, it triggers the prefetcher and clogs up it up even if it only needs to read parts of an entire file.

 

 

I don't know what to tell you. It does function like that. Please see the documentation at https://stablebit.co...I/O Performance. Your concerns are also misguided. It is a maximum window, not a required window. If it accesses 1MB during any time *less* than the window it will trigger the prefetch. So 1MB in 30secs is very easy to hit. An 8 mbps video file would trigger the prefetch in one second with a 1MB trigger; a 4mbps video file would trigger it in 2 secs. 

 

 

 

Do you have any proof that the Transcoder default throttle buffer is CPU time? "Amount in seconds to buffer before throttling back the transcoder speed." is what the Plex site says and I can't find any other sources other than suggested amounts on the Rclone forums of 10 minutes. Even if it is CPU time, it tells Plex to keep trancoding additional data and the Plex client will buffer and cache it.

 

 

You can see a Plex employee explain the setting here: https://forums.plex....throttle-buffer

 

It is, in any case, not a necessary setting to modify if your CloudDrive settings are correct. CloudDrive is plenty responsive enough, particularly on a 1gbps connection, to download the necessary data in chunks without affecting your streaming. Let the server throttle down to make room for another stream. It'll unthrottle when it needs to regardless, so you're not really gaining anything.  

 

 

 

In any case, I am certainly getting better performance with these settings. I can see spikes of data streaming / trancoding when the large prefetch gets downloaded and then processed by Plex and sent to the client. I watched 4x 1080p streams get handled flawlessly. That being said, I do have gigabit and 20x 2 GHz cores on the server.

 

 

You'd likely get even more performance if you optimized things correctly. My server is only an E3 1245v2 and it easily handles 5-8 simultaneous streams on a daily basis, and most of my media is stored in Remux quality (20-35mbps or so). Yours should be able to handle far, far more than 4 1080p streams with your specs. 

 

I suggest these settings for Plex based on a lot of testing and experimentation:

 

Download threads: 10

Upload threads: 5

Upload threshold: Irrelevant, really, but 1MB/5mins

Minimum Download Size: 10MB

Prefetch trigger: 1MB

Prefetch Forward: 150MB

Prefetch time window: 30secs

 

20MB storage chunks on gdrive, largest RAM chunk cache and expandable cache you can accommodate with your setup. 

 

 

That should provide you with a responsive drive with more than enough speed to stream basically anything you throw at it. 

 

EDIT: Just saw that Christopher responded at the same time. His suggestion of a 10MB trigger is probably fine too. I'd imagine 1MB or 10MB makes little difference with high quality media. You want it to prefetch often for Plex, in any case. 



#7 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 13 October 2017 - 07:55 PM

thanks guys, that all makes sense.

 

I still think there's something funky with the "Prefetched:" figure in the UI then. Seriously, try modifying the the Prefetch Time Window to something like 10 seconds, then prefetch something. That number will go up to the amount that you prefetched (150 MB or whatever), then drop down to 0 MB (and disappear), 10 seconds later. Now set it to 300 seconds and prefetch. It'll keep that data for 300 seconds, then drop down to 0 MB (and disappear).

 

so either there's a bug in the UI, the documentation is wrong, or the functionality is different.



#8 srcrist

srcrist

    Advanced Member

  • Members
  • PipPipPip
  • 89 posts

Posted 13 October 2017 - 10:19 PM

I think the prefetch functionality remains a little unclear in the documentation. I'd love for Alex to give an account of how it works in more detail as well. It would help a lot with respect to optimization. One question that I still have is whether or not defragmenting the drive would help the prefetcher to obtain contiguous data while streaming. Based on my current understanding of how the process works, I think it would. But that's still a bit unclear. Perhaps Christopher could ask for a nuts and bolts post on the feature?



#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,394 posts
  • LocationSan Diego, CA, USA

Posted 14 October 2017 - 12:53 AM

thanks guys, that all makes sense.

 

I still think there's something funky with the "Prefetched:" figure in the UI then. Seriously, try modifying the the Prefetch Time Window to something like 10 seconds, then prefetch something. That number will go up to the amount that you prefetched (150 MB or whatever), then drop down to 0 MB (and disappear), 10 seconds later. Now set it to 300 seconds and prefetch. It'll keep that data for 300 seconds, then drop down to 0 MB (and disappear).

 

so either there's a bug in the UI, the documentation is wrong, or the functionality is different.

 

 

The reason for that prefetching is that the trigger has to occur over the course of the time window.    Setting the window too low, or the trigger too high means that it's not as likely to trigger the prefetching.   Which is what you were seeing. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 Soaringswine

Soaringswine

    Member

  • Members
  • PipPip
  • 16 posts

Posted 14 October 2017 - 12:57 AM

The trigger is happening regardless, it's the "Prefetched" amount that stays around longer, indicating to me that the prefetched data is being kept in the prefetch cache longer. this also means you can easily accrue 10-30 GB of prefetch cache if you set your window high.

 

try my experiment that I stated above, you'll see.



#11 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,394 posts
  • LocationSan Diego, CA, USA

Posted 16 October 2017 - 05:09 PM

As long as you have the space in the cache, it will keep the prefetched data. 

 

Also, older (stale) data may be pushed out of the cache, depending. 

 

Otherwise, even with the default settings, I see it prefetching, and maintaining a small amount of data (roughly 70MBs) is it's reading a number of smaller files. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users