Question regarding larger chunk sizes

[modplan]

I've currently settled on chunk sizes of 50MB as being the best performing for me at the moment, but have a question.

 

If I create a 1MB file on my drive, what gets uploaded? A 50MB object that is 49MB of zeroes or do we cache until we see 50MB of new data and then upload?

[Christopher (Drashna)]

I believe that all chunk sizes larger than 1MB are partial upload and downloads.  That means only the specific section of data should be uploaded.  So ~1MB of data, rather than all 50MB.

[modplan]

Weird. It appears, if I watch the IO threads, if I create a 2.5MB file and "Too Upload" says about 2.5MB, the IO threads report uploading an entire 50MB file.

[Christopher (Drashna)]

If you haven't already, upgrade to the 421 build.  Alex has massively redesigned the chunk cache stuff.  \

 

 

* For the Local Disk provider, checksumming can now be optionally enabled for chunks up to 100 MB in size.

* Chunks up to 100 MB in size are now enabled for:

    - Amazon Cloud Drive (experimental)

    - Amazon S3

    - Box

    - Dropbox

    - Google Cloud Storage

    - Google Drive

    - Microsoft Azure Storage

    - OneDrive (experimental)

* Added a new checksumming algorithm and chunk I/O pipeline that supports partial read checksumming of read unit sizes. This works 

  together with large chunk sizes (> 1 MB) and partial reads (for providers that support that) and allows for checksumming to take 

  place with large chunks.

    - This checksumming algorithm (CRC32-Block) supersedes the old (CRC32) algorithm and will be used for all new drives by default.

    - The old algorithm is still supported and will work transparently for older drives (that were limited to 1 MB chunks).

 

 

This may completely changed how the upload is handled

[steffenmand]

you guys are awesome! and really listening to the comments in here!