Cache causing very high disk usage

[Trasher]

I'm in the process of uploading 15TB of data to my cloud drive, however it's taking quite a long time because the cloud drive service and system(NT Kernel) are causing 99% disk usage and windows slows down to a crawl

I'm using Goodsync to backup from a Google drive to a Stablebit drive (also in Google drive), this is on a dediserver with 2xHDDs in RAID0 and 1GB connection

Downloading / Uploading to google drive itself is alot faster (70-75MB/s)..with stablebit I'm getting 6-8MB/s? 

[Christopher (Drashna)]

Depending on the situation, this may be normal.  The cache is IO intensive, especially if there is a lot of activity hitting the drive. 

 

 

As for the performance, there are a few things that can affect this.   Unfortunately, some of these may increase disk activity, actually. 

 

• Increasing the number of upload threads may help. it means more chunks are uploaded at once, but this will increase disk activity. 
• Disable the prefetcher
• If you don't have a lot of "to upload" data, detach the drive and reattach it. Set the "minimum download size" to 5-7MBs, and then attach (this is in the advanced settings while attaching the drive). 

[Trasher]

Thanks,

I have Sonarr/Radarr/Plex pointed to the Stablebit drive and with a little bit of tweaking it seems to be working, i reduced UL threads to 3, as this uploads almost as fast as the cache is filling up.

And anyways, it will all be uploaded in a month or so

[fishie]

 

Depending on the situation, this may be normal.  The cache is IO intensive, especially if there is a lot of activity hitting the drive. 

 

 

As for the performance, there are a few things that can affect this.   Unfortunately, some of these may increase disk activity, actually. 

 

• Increasing the number of upload threads may help. it means more chunks are uploaded at once, but this will increase disk activity. 
• Disable the prefetcher
• If you don't have a lot of "to upload" data, detach the drive and reattach it. Set the "minimum download size" to 5-7MBs, and then attach (this is in the advanced settings while attaching the drive). 

 

 

Christopher, im actually fairly confident that there is a problem here.

 

I have been troubleshooting this, also in correspondance to the other thread I have posted in, and whenever writes are going to the clouddrive cache it is slow as hell - Recently I picked up some new Samsung 960Pro nvme drives, thought it would be fun to test them out so I put 2 of them together in a raid0 and used them strictly for cache for the clouddrive, that should eliminate any kind of performance issues as those drives are basically able to write in the GB\s range - But im still seeing subpar performance as in writes to the clouddrive cache is below 10MB\s with it being fully maxed out according to windows.

 

Im not sure if it is something to do with the settings or a software problem, I can simply let you know that the slowdowns are really odd, to the point that faster drives provide no faster throughput on the cache drive, the two 960Pro drives provided me with ~1MB\s more than my single Kingston SSD I have been using in the past.

 

It has come to the point where I have simply uninstalled the clouddrive software, and now use rclone to move my data to google instead as that is about 15 times faster.

[Christopher (Drashna)]

Christopher, im actually fairly confident that there is a problem here.

 

I have been troubleshooting this, also in correspondance to the other thread I have posted in, and whenever writes are going to the clouddrive cache it is slow as hell - Recently I picked up some new Samsung 960Pro nvme drives, thought it would be fun to test them out so I put 2 of them together in a raid0 and used them strictly for cache for the clouddrive, that should eliminate any kind of performance issues as those drives are basically able to write in the GB\s range - But im still seeing subpar performance as in writes to the clouddrive cache is below 10MB\s with it being fully maxed out according to windows.

 

Im not sure if it is something to do with the settings or a software problem, I can simply let you know that the slowdowns are really odd, to the point that faster drives provide no faster throughput on the cache drive, the two 960Pro drives provided me with ~1MB\s more than my single Kingston SSD I have been using in the past.

 

It has come to the point where I have simply uninstalled the clouddrive software, and now use rclone to move my data to google instead as that is about 15 times faster.

 

 

That's definitely unusual, then. 

 

However, antivirus software could be the culprit here. 

 

Otherwise, if you wanted to reinstall the software:

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

 

And using the "Windows Performance Recorder" may be useful in telling what is going on here.  

 

Otherwise, enjoy RCLONE, as it does work well. 

[fishie]

@Christopher,

 

Im sorry for the long wait here, I have been busy with work and have not had the chance to tinker with this.

 

Im actually on a diffent OS now, so I will have to test this out again in the future.

 

Is there any pointers on how to optimize the cache setup? I tend to go for 100MB chunks but is it really faster to use smaller chunks with a smaller cache set for instance? I would rather not use rclone to be honest as it is a "hack" on windows and would love to run something that natively supports Windows, especially since I already bought the software

[Edrock200]

In addition to av, you'll also want to turn off windows indexing on the hidden cache drive directory

[Christopher (Drashna)]

@Christopher,

 

Im sorry for the long wait here, I have been busy with work and have not had the chance to tinker with this.

 

Im actually on a diffent OS now, so I will have to test this out again in the future.

 

Is there any pointers on how to optimize the cache setup? I tend to go for 100MB chunks but is it really faster to use smaller chunks with a smaller cache set for instance? I would rather not use rclone to be honest as it is a "hack" on windows and would love to run something that natively supports Windows, especially since I already bought the software

 

 

use smaller chunks.   For both options.    

 

Also, in the advanced settings when creating or attaching the drive, set the "minimum download size".

 

Tinker with the number of upload and download threads. 

 

 

 

These should help