Request: Increased block size

[steffenmand]

Recently got 10 gbit internet with my server, however are seeing that the maximum size of 20 MB in blocks are causing speeds to be limited around 40 MB/s due to the response time delays pr. thread

Would love to request that it gets possible to increase the block size further, to acommodate increasing internet speeds over the coming years.

maybe a 50 MB block size and ofc. a warning showcasing that high internet speeds are required (just like with the minimal download size)

[srcrist]

3 hours ago, steffenmand said:

Recently got 10 gbit internet with my server, however are seeing that the maximum size of 20 MB in blocks are causing speeds to be limited around 40 MB/s due to the response time delays pr. thread

Would love to request that it gets possible to increase the block size further, to acommodate increasing internet speeds over the coming years.

maybe a 50 MB block size and ofc. a warning showcasing that high internet speeds are required (just like with the minimal download size)

The maximum chunk size is actually a per-provider limitation. Some providers *can* use chunks larger than 20MB. During Beta, Google could use chunks as large as 100MB, if I remember right, but that caused some sort of issue, which escapes me, with Google's service and API limitations. So this isn't really a matter of CloudDrive's features, but those supported by the provider you're using. 

[steffenmand]

On 4/11/2019 at 7:16 PM, srcrist said:

The maximum chunk size is actually a per-provider limitation. Some providers *can* use chunks larger than 20MB. During Beta, Google could use chunks as large as 100MB, if I remember right, but that caused some sort of issue, which escapes me, with Google's service and API limitations. So this isn't really a matter of CloudDrive's features, but those supported by the provider you're using. 

We never had bigger chunks than 20 MB (me who early on pushed for higher blocks than 1 MB) EDIT: Or actually we did, but they removed it within no time. If it really was a provider issue, it should be an easy "if that then that" solution as they know what provider we are building drives on

Their worries was the drive being not responsive as chunks would take time to download - which wouldnt be the case with the right setup

Just seeing the limitations now from using that block size as both servers and internet get faster as well as the data getting bigger and bigger

[srcrist]

17 hours ago, steffenmand said:

We never had bigger chunks than 20 MB (me who early on pushed for higher blocks than 1 MB) EDIT: Or actually we did, but they removed it within no time. If it really was a provider issue, it should be an easy "if that then that" solution as they know what provider we are building drives on

 

Again, other providers *can* still use larger chunks.

Please see the changelog:

Quote

.421
* For the Local Disk provider, checksumming can now be optionally enabled for chunks up to 100 MB in size.
* Chunks up to 100 MB in size are now enabled for:
    - Amazon Cloud Drive (experimental)
    - Amazon S3
    - Box
    - Dropbox
    - Google Cloud Storage
    - Google Drive
    - Microsoft Azure Storage
    - OneDrive (experimental)

.450
* [Issue #24914] Do not allow chunk sizes > 20 MB for any newly created cloud drives using the Google Drive provider.

This was because of issue 24914, documented here.

17 hours ago, steffenmand said:

Their worries was the drive being not responsive as chunks would take time to download - which wouldnt be the case with the right setup

Just seeing the limitations now from using that block size as both servers and internet get faster as well as the data getting bigger and bigger

Again, this isn't really correct. The problem, as documented above, is that larger chunks results in more retrieval calls to particular chunks, thus triggering Google's download quota limitations. That is the problem that I could not remember. It was not because of concerns about the speed, and it was not a general problem with all providers. 

 

EDIT: It looks like the issue with Google Drive might be resolved with an increase in the partial read size as you discussed in this post, but the code change request for that is still incomplete. So this prerequisite still isn't met. Maybe something to follow up with Christopher and Alex about. 

[steffenmand]

21 hours ago, srcrist said:

Again, other providers *can* still use larger chunks.

Please see the changelog:

This was because of issue 24914, documented here.

Again, this isn't really correct. The problem, as documented above, is that larger chunks results in more retrieval calls to particular chunks, thus triggering Google's download quota limitations. That is the problem that I could not remember. It was not because of concerns about the speed, and it was not a general problem with all providers. 

 

EDIT: It looks like the issue with Google Drive might be resolved with an increase in the partial read size as you discussed in this post, but the code change request for that is still incomplete. So this prerequisite still isn't met. Maybe something to follow up with Christopher and Alex about. 

Look at my Edit, i did acknowledge it was there - remember so as well - but they removed in an update or two later (updates almost every other day back then).

Yes, but that was with a use-case of partial downloads in mind. I do remember them highlighting issues with responsiveness as being the reason why we couldn't go above 20 MB, but the limitations of max amount of reads pr. day. pr file. is of course a valid argument as well. Was unaware that they kept it for the remaining providers.

Remember saying that those block sizes should be intended for high bandwidth use ie. minimal download of 50 MB-> 100MB. Still think that is the case today.

In general the desire is to allow high bandwidth to utilize speeds. The bottleneck today is the responsetime pr. thread causing speed to massively drop!

A good solution is 100 MB chunks = minimal download atleast of 20 MB, again a lot of "if that then that"

I will reach out to Christopher about getting my old feature back on track

[Christopher (Drashna)]

On 4/11/2019 at 6:06 AM, steffenmand said:

Recently got 10 gbit internet with my server, however are seeing that the maximum size of 20 MB in blocks are causing speeds to be limited around 40 MB/s due to the response time delays pr. thread

 Would love to request that it gets possible to increase the block size further, to acommodate increasing internet speeds over the coming years.

 maybe a 50 MB block size and ofc. a warning showcasing that high internet speeds are required (just like with the minimal download size)

There is an issue here, though.  With Google Drive, there is a hard limit on the number of times a file can be accessed.  That's why Google Drive is limited to 20MB.  So, if we allow a larger size, we would have to enforce a minimum download size proportional to the chunk size. 

 

 

 

[steffenmand]

5 hours ago, Christopher (Drashna) said:

There is an issue here, though.  With Google Drive, there is a hard limit on the number of times a file can be accessed.  That's why Google Drive is limited to 20MB.  So, if we allow a larger size, we would have to enforce a minimum download size proportional to the chunk size. 

 

 

 

and this is what would fix it! people using 100 MB chunks are usually high bandwidth users, so forcing minimum download of 20 MB or more should be fine! right now we just have a bottleneck with the response times slowing down downloads.

i would go for for 100 mb downloads anyway

 

[Bowsa]

I have symmetrical 1000 gbps, and have seen my downloads reach 850-960 mbps. Are you set-up correctly with the threads?

[steffenmand]

On 4/16/2019 at 5:26 PM, Bowsa said:

I have symmetrical 1000 gbps, and have seen my downloads reach 850-960 mbps. Are you set-up correctly with the threads?

i think you mean mbit :-P

Yes. It all depends on the response time you have. Speed is not the issue, it's my response time to google's servers You're just lucky to be closer.

Plus i got upload verification on, that also cuts off speeds on uploads  I get around 2500-2800 ms response time pr. thread and then instant download. So the less calls and the bigger download would do wonders for me