Is there any way to speed up downloads from my CloudDrive?

[pedges]

I have about 5TB worth of data in a CloudDrive (via Google Drive) that I'm trying to copy to a physical hard drive and it is taking forever to do so.

I have a gigabit internet connection, so I don't think that is affecting CloudDrive's ability to download data to be able to move it.

Is there a faster way to do this that I'm not aware of? So far it's been running ~12 hours and has only been able to move ~30GB to the physical hard drive.

[steffenmand]

26 minutes ago, pedges said:

I have about 5TB worth of data in a CloudDrive (via Google Drive) that I'm trying to copy to a physical hard drive and it is taking forever to do so.

I have a gigabit internet connection, so I don't think that is affecting CloudDrive's ability to download data to be able to move it.

Is there a faster way to do this that I'm not aware of? So far it's been running ~12 hours and has only been able to move ~30GB to the physical hard drive.

Did you do big chunks when creating the drive ?

 

Also remember to increase the thread count in I/O operations.

[pedges]

1 minute ago, steffenmand said:

Did you do big chunks when creating the drive ?

 

Also remember to increase the thread count in I/O operations.

I think I set it to the largest chunk size when I created the drive. And I'll try increase the thread count and see if that has any impact at all.

[pedges]

5 minutes ago, steffenmand said:

Did you do big chunks when creating the drive ?

 

Also remember to increase the thread count in I/O operations.

I increased download threads to 4 (it was at 2) and minimum download size to 40MB.

Should I increase those two even further, or how do I know the limit to which I can increase them to?

[steffenmand]

5 minutes ago, pedges said:

I increased download threads to 4 (it was at 2) and minimum download size to 40MB.

Should I increase those two even further, or how do I know the limit to which I can increase them to?

The more threads, the more it downloads at the same time, thus faster speed. However of course you should not do more threads than your CPU can handle - as well there can be an upper limit from the cloud provider for max api calls.

I use 14 threads

[pedges]

23 hours ago, steffenmand said:

The more threads, the more it downloads at the same time, thus faster speed. However of course you should not do more threads than your CPU can handle - as well there can be an upper limit from the cloud provider for max api calls.

I use 14 threads

Is there an obvious way to tell how many threads my CPU can handle? I'm up to 10 threads and 100MB minimum download and it's still taking forever to download even smaller amounts of data.

This is the server I use: https://www.ebay.com/itm/Dell-Precision-T5600-16-Core-2-20GHz-E5-2660-16GB-RAM-2x-500GB-HDD/182388025949?hash=item2a772c6a5d:g:B4YAAOSw3Z9asP2S

[steffenmand]

8 hours ago, pedges said:

Is there an obvious way to tell how many threads my CPU can handle? I'm up to 10 threads and 100MB minimum download and it's still taking forever to download even smaller amounts of data.

This is the server I use: https://www.ebay.com/itm/Dell-Precision-T5600-16-Core-2-20GHz-E5-2660-16GB-RAM-2x-500GB-HDD/182388025949?hash=item2a772c6a5d:g:B4YAAOSw3Z9asP2S

my guess is that you made the drive with 1 mb chunks.

your cpu has 16 cores and can run up to 32 threads total. so 14 threads should be fine.

for high speeds always remember to do large chunk drives - else you cant get proper speeds.

[srcrist]

The threads in CloudDrive are not CPU threads, and are not CPU limited. They are just simultaneous upload and download requests. They are limited by frequency and your provider's API limitations. Getting the correct value, for you, will be a matter of experimentation. I suggest starting with 10 download threads and 5 upload, and adjusting from there. You should *never* need more than a handful of upload threads, in any case, unless your network is misconfigured. Even 2 or 3 upload threads is capable of hitting the 70-75mpbs throughput that Google allows--permitting that your connection speed is capable of doing so. See the manual page here for more information. 

11 hours ago, pedges said:

Is there an obvious way to tell how many threads my CPU can handle? I'm up to 10 threads and 100MB minimum download and it's still taking forever to download even smaller amounts of data.

 

This is the exact opposite of what you want. That's why. The larger your minimum download, the less responsive your drive will be. A 100 MB minmum download means that even if an application only requests 1 MB of data, CloudDrive is still downloading 100MB. There is effectively no reason for your minimum download to ever be larger than your chunk size, for most use cases. Raising your minimum is actually working against you here. Go with my above recommendations for thread count, and drop the minimum download to somewhere between 10 and 20 MB. See how that performs. We can make adjustments to your prefetcher to accommodate larger data requests over and above that amount. 

 

EDIT: Also disable Background IO on the drive in the performance settings if read response is important.  

[steffenmand]

6 hours ago, srcrist said:

The threads in CloudDrive are not CPU threads, and are not CPU limited. They are just simultaneous upload and download requests. They are limited by frequency and your provider's API limitations. Getting the correct value, for you, will be a matter of experimentation. I suggest starting with 10 download threads and 5 upload, and adjusting from there. You should *never* need more than a handful of upload threads, in any case, unless your network is misconfigured. Even 2 or 3 upload threads is capable of hitting the 70-75mpbs throughput that Google allows--permitting that your connection speed is capable of doing so. See the manual page here for more information. 

This is the exact opposite of what you want. That's why. The larger your minimum download, the less responsive your drive will be. A 100 MB minmum download means that even if an application only requests 1 MB of data, CloudDrive is still downloading 100MB. There is effectively no reason for your minimum download to ever be larger than your chunk size, for most use cases. Raising your minimum is actually working against you here. Go with my above recommendations for thread count, and drop the minimum download to somewhere between 10 and 20 MB. See how that performs. We can make adjustments to your prefetcher to accommodate larger data requests over and above that amount. 

 

EDIT: Also disable Background IO on the drive in the performance settings if read response is important.  

srcrist.

Nobody said it was CPU threads. HOWEVER your CPU has a limitation as to how many threads it can run as a system at a time - thus putting it higher than the CPU capacity will only build queues. Not comparing it to the same type of threads it still means a limitation in concurrent operations.

 

Also faily certain that the "minimal download size" will never download anything larger than the chunk size. So if the chunk size is set to 1 MB, then 1 MB is the max download. Im pretty sure that the feature was made for partial downloads of data from larger chunks - feel free to correct me here.

[srcrist]

52 minutes ago, steffenmand said:

Nobody said it was CPU threads. HOWEVER your CPU has a limitation as to how many threads it can run as a system at a time - thus putting it higher than the CPU capacity will only build queues. Not comparing it to the same type of threads it still means a limitation in concurrent operations.

 

This is incorrect. Adjusting the number of threads in CloudDrive has no appreciable impact on CPU usage. Threads, in this case, simply refers to the number of simultaneous upload and download requests that the application will let itself make at any given time. CloudDrive has limits on the number of threads that one can set in the application, so it isn't possible to test whether or not thousands of threads may eventually lead to a degradation of CPU performance, but, under the limitations that exist today, you will hit the API limitations of the provider long before any impact on CPU usage is worthy of consideration. In short: Do not adjust CloudDrive's threads based on CPU related concerns, and there is no relationship between "threads" as related to processor scheduling and "threads" as they are used in this context--other than the name.

57 minutes ago, steffenmand said:

Also faily certain that the "minimal download size" will never download anything larger than the chunk size. So if the chunk size is set to 1 MB, then 1 MB is the max download. Im pretty sure that the feature was made for partial downloads of data from larger chunks - feel free to correct me here.

As far as I know, this is also incorrect. A minimum download size larger than the chunk size will still download multiple chunks of contiguous data until the specified threshold is met, again, as far as I know. There is nothing conclusive in the documentation about exactly how this functions, but I have seen Christopher recommend settings higher than 20MB for Google Drive, such as he did in this case and this case, for specific situations. That would lead me to believe that higher settings do, in fact, have an impact beyond a single chunk. This is, of course, with the caveat that, as explained in the user manual, CloudDrive can still request less data than the "minimum" if the system is also requesting less (total) data.

In any case, it is true, either way, that a larger minimum download will reduce the responsiveness of the drive, and that speeds in excess of 600-700mbps can be achieved with a minimum download of 10-20MB using the settings I recommended above. So if the problem is responsiveness, dropping it may help. 

The base settings that I would recommend anyone try if they are having performance issues are the following, and then play with them from there:

• 5 Upload Threads
• 10 Download Threads
• Upload Throttling enabled at approx 70mbps (to stay within Google's 750GB/day limitation)
• Upload Threshold 1MB/5 Mins
• Minimum Download 10-20MB
• Disable Background IO if you'll be streaming media so that downstream performance is not impacted by upstream requests. 

If that works, you're great, and, if not, then you come back and people can try to help. But these are settings that have worked for a lot of people here over the last several years. 

[steffenmand]

2 hours ago, srcrist said:

This is incorrect. Adjusting the number of threads in CloudDrive has no appreciable impact on CPU usage. Threads, in this case, simply refers to the number of simultaneous upload and download requests that the application will let itself make at any given time. CloudDrive has limits on the number of threads that one can set in the application, so it isn't possible to test whether or not thousands of threads may eventually lead to a degradation of CPU performance, but, under the limitations that exist today, you will hit the API limitations of the provider long before any impact on CPU usage is worthy of consideration. In short: Do not adjust CloudDrive's threads based on CPU related concerns, and there is no relationship between "threads" as related to processor scheduling and "threads" as they are used in this context--other than the name.

As far as I know, this is also incorrect. A minimum download size larger than the chunk size will still download multiple chunks of contiguous data until the specified threshold is met, again, as far as I know. There is nothing conclusive in the documentation about exactly how this functions, but I have seen Christopher recommend settings higher than 20MB for Google Drive, such as he did in this case and this case, for specific situations. That would lead me to believe that higher settings do, in fact, have an impact beyond a single chunk. This is, of course, with the caveat that, as explained in the user manual, CloudDrive can still request less data than the "minimum" if the system is also requesting less (total) data.

In any case, it is true, either way, that a larger minimum download will reduce the responsiveness of the drive, and that speeds in excess of 600-700mbps can be achieved with a minimum download of 10-20MB using the settings I recommended above. So if the problem is responsiveness, dropping it may help. 

The base settings that I would recommend anyone try if they are having performance issues are the following, and then play with them from there:

• 5 Upload Threads
• 10 Download Threads
• Upload Throttling enabled at approx 70mbps (to stay within Google's 750GB/day limitation)
• Upload Threshold 1MB/5 Mins
• Minimum Download 10-20MB
• Disable Background IO if you'll be streaming media so that downstream performance is not impacted by upstream requests. 

If that works, you're great, and, if not, then you come back and people can try to help. But these are settings that have worked for a lot of people here over the last several years. 

Minimal download size above 20 MB on google drive makes no difference on my system

[pedges]

For what it's worth, I've been using CloudDrive to store my Plex media for the past year and the default settings have always worked well for me.

The reason I posted this question is because I'd like to move all my data out of and away from CloudDrive. After the GDrive corrupt data issue a few weeks ago, I lost a ton of media on one of my drives and no longer trust it being in the cloud, so I'm trying to do a bulk download to physical hard drives.

The default settings do not seem catered to performing bulk downloads, which is why we've been messing with all of the I/O options to figure out what would.

From what I've found so far, the fastest combination of settings seems to be 1MB trigger/20MB forward/30 second time window for the prefetch settings, with 14 download threads and a 20MB minimum download size. I'm fairly certain I maxed out the chunk size when creating the drive, but I really can't remember.

It's still not blazing fast, but it is much faster than the default settings I was using just two days ago.

[srcrist]

5 hours ago, pedges said:

I'm fairly certain I maxed out the chunk size when creating the drive, but I really can't remember.

You can actually check this, in the UI, by mousing over the drive size at the top under the provider logo. If you want to. 

[Knxwledge]

9 hours ago, srcrist said:

You can actually check this, in the UI, by mousing over the drive size at the top under the provider logo. If you want to. 

Does Chunk Size really matter? I have mine set to 10 but I see everyone setting it to 20. Is 20MB better? Am I able to change the chunk size of a drive?

[srcrist]

6 hours ago, Knxwledge said:

Does Chunk Size really matter? I have mine set to 10 but I see everyone setting it to 20. Is 20MB better? Am I able to change the chunk size of a drive?

That isn't a question with an objective, universal answer. The benefits to different chunk sizes vary depending on what you want to use your drive for. A lot of people use their CloudDrive drives to store large media libraries. So the maximum chunk size for Google, which is 20MB, allows for the highest capacity drive, and the highest throughput for streaming media off of the drive. 10MB chunks are *also* probably fine for this purpose, but the drive cannot be as large (though it can still be huge), and the theoretical maximum downstream speed will be lower. 

You cannot, in any case, change the chunk size of a drive once it has been created. That's one of the settings that has to be chosen upon drive creation, and changing it requires a fundamental change in the structure of the drive as it is stored on the provider, so it can't be changed later.