Amazon Cloud Drive storage chunk size

[612WharfAvenue]

I've been testing out CloudDrive with Amazon cloud drive over the past couple days, and have run into the "Server is throttling us" log message with some frequency (alongside "Thread was being aborted")   I noticed that the maximum storage chunk size for the ACD provider is 1MB, as opposed to much higher elsewhere.      It seems the throttling is based on transaction volume rather than data size, as I'm able to upload multi-gig test files at full speed with no errors. 

 

Is there a technical reason for the 1MB limit?     It seems that a larger chunk size would reduce transaction count for the same amount of data and drop overhead, possibly allowing moving a large file to an ACD-backed CloudDrive to approach a raw upload in terms of speed.

[Christopher (Drashna)]

1MB is the chunk size we use.  The files are stored on the provider as 1MB files.

 

This way, we can rapidly grab a chunk of data, without having to download a huge section of data.  Or so it doesn't take forever to upload the chunk because you added a simple text file.

 

 

This setting can be changed when you create the drive. Just click on the Advanced settings section.  This should show the additional settings. Find the "Storage chunk size" and set this to a higher value.

 

 

 

Just note, that Amazon Cloud Drive *requires* upload verification, so every chunk you uploaded will be downloaded again to verify that it uploaded properly. Larger chunks means more data is transferred each way, and may take longer and impact performance.

 

 

 

And you may see why we chose 1MB as the default size here. It's a good size, that allows it to get a good chunk of data, without being too large.

[612WharfAvenue]

 Find the "Storage chunk size" and set this to a higher value.

 

 

 

Just note, that Amazon Cloud Drive *requires* upload verification, so every chunk you uploaded will be downloaded again to verify that it uploaded properly. Larger chunks means more data is transferred each way, and may take longer and impact performance.

 

 

 

And you may see why we chose 1MB as the default size here. It's a good size, that allows it to get a good chunk of data, without being too large.

 

 

 

 

 

I did find the chunk size setting, and as noted in my initial post, it's capped at 1MB for ACD.    If it had a larger range of options I could experiment and find the ideal tradeoff between throughput for large files and latency.   The issue I'm running into (and I suspect anyone wiht a higher bandwidth upload is) is that Amazon is throttling things, seemingly on transaction count.    Larger chunk size would mitigate this, albeit at the cost of requiring a minimum >1MB read to pull any amount of data down.      Upload verification seems like it would require both uploading and downloading all data, whether a theoretical 500MB file is divided into 1MB or 10MB chunks, 500MB is going up and coming down - the only tradeoff would be a bigger penalty for a chunk that fails upload verification - this would seem a small price to pay to avoid throttling, however.

[Christopher (Drashna)]

Okay, I'll flag Alex about larger chunk sizes.

 

But in the meanwhile, once the drive is created and attached, click on "Disk Options", open the "performance" section and select "IO Performance".  Lower the read and write threads.  Set it to 1 for both, for now, and then increase the amount of threads if it's not having issues.

 

 

Larger chunk size would mitigate this, albeit at the cost of requiring a minimum >1MB read to pull any amount of data down.      Upload verification seems like it would require both uploading and downloading all data, whether a theoretical 500MB file is divided into 1MB or 10MB chunks, 500MB is going up and coming down - the only tradeoff would be a bigger penalty for a chunk that fails upload verification - this would seem a small price to pay to avoid throttling, however.

 

Well, StableBit CloudDrive doesn't know about the actual files on the file system. Just the RAW blocks. That's what we upload. 

[Kraevin]

I have been trying to set this up with Amazon Cloud drive but no matter what i do, i still get a ton of I/O errors saying its bandwidth issues. I am still using the trial version, does the trial have a bandwidth limit set in place? or is it just that bad with amazon cloud? 

 

I have tried adjusting the settings as stated above and also changing the read and write to even 1 and get the i/o errors.

[Christopher (Drashna)]

We apologize for the issue.

 

This is mostly our fault.

 

Specifically, we need to get "production validation" so we can get off of just developer authorization.  Otherwise, we're faced with a rate limit for upload and download speeds, and amount of traffic that is permitted.  Since the documentation didn't really make this clear.... We didn't do this until after the product was released publicly.  Additionally, currently the portal for getting the authorization/validation is apparently down.  We're doing what we can to fix this, so that the provider is stable.  But in the meanwhile, we'd recommend against using Amazon Cloud Drive as a provider.

[Kraevin]

Appreciate the update and the explanation, glad to see you guys are working on the issue. Please keep us  updated on the updates =)

[Christopher (Drashna)]

We're still waiting on Amazon. Still having issues with their dashboard, and no response from emails (I'd guess that they're having a major issue in their backend right now... which wouldn't be surprising).

 

However, we've added some better handling to the throttling messages we get from Amazon, and we have a much improved cache system, that should fix any issues we were seeing with it. 

 

If you're interested in it:

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.330_x64_BETA.exe

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.330_x86_BETA.exe

[Kraevin]

We're still waiting on Amazon. Still having issues with their dashboard, and no response from emails (I'd guess that they're having a major issue in their backend right now... which wouldn't be surprising).

 

However, we've added some better handling to the throttling messages we get from Amazon, and we have a much improved cache system, that should fix any issues we were seeing with it. 

 

If you're interested in it:

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.330_x64_BETA.exe

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.330_x86_BETA.exe

 

Awesome! I will test it out =) 

 

Thanks!

[Christopher (Drashna)]

I've edited the links, as the previous links, the installers has some issues performing an upgrade properly. The 330 build should be fine though.

[Kraevin]

Huge improvement so far with the update in regard to the tons of i/o errors we were getting, looking forward too see what's to come =)

[Christopher (Drashna)]

Glad to hear it!

[WBadminUser]

I believe this issue should be a little more clear. (A few months back) I used the product and was like man its trash with all these errors its unusable. Decided to try to figure out why this was happening and was going to do my own post but decided to spend 30 minutes reading the forums first to find the answers. While i am really glad the issues are supposedly fixed. it just seems like a lot of users would not have gone through this trouble.

 

Also did you ever get on the amazon support list?

[Christopher (Drashna)]

I believe this issue should be a little more clear. (A few months back) I used the product and was like man its trash with all these errors its unusable. Decided to try to figure out why this was happening and was going to do my own post but decided to spend 30 minutes reading the forums first to find the answers. While i am really glad the issues are supposedly fixed. it just seems like a lot of users would not have gone through this trouble.

 

Also did you ever get on the amazon support list?

It's not exactly clear what you mean here....

 

 

However, the deadlock issue was the main issue we were seeing. It was occurring in several places, actually. 

Amazon Cloud Drive was definitely making the issue worse (due to its frequent errors, and slow speeds).  

We do plan on pushing out an update to everyone very, very soon, though.

 

 

As for the support list, as far as I'm aware, none of our emails have been responded to.  And for the production stuff.... we were briefly approved, but we had a single user connecting at ~50MB/s and Amazon decided to switch our entire product back to dev status rather than throttling the one user.  It's a "charlie fox", and to be honest, I'm not sure Amazon Cloud Drive is a reliable provider at this point.  Alex gets the final call on that one, but .... I'd recommend against it, personally.

[WBadminUser]

I just meant that you guys are not very front facing with amazon cloud drive issues.

Last night i installed the 330 beta from up the thread and it was still unusable(I had hopes it was resolved from reading above). I am sure its pretty complicated but i have used 4 different products that connect to amazon cloud drive just fine. I will refrain from listing them here though. I find it a stretch to blame amazon on this one. I really hope you guys get it working as its a nice combo of pieces(Chunking encryption etc) that i could find myself using if it just worked.

[Kraevin]

I just meant that you guys are not very front facing with amazon cloud drive issues.

Last night i installed the 330 beta from up the thread and it was still unusable(I had hopes it was resolved from reading above). I am sure its pretty complicated but i have used 4 different products that connect to amazon cloud drive just fine. I will refrain from listing them here though. I find it a stretch to blame amazon on this one. I really hope you guys get it working as its a nice combo of pieces(Chunking encryption etc) that i could find myself using if it just worked.

 

the .330 is a very old version

 

try .378, while it still shoots out a couple errors, but it performs a lot better than previous versions.

 

http://dl.covecube.com/CloudDriveWindows/beta/download/

[Christopher (Drashna)]

As for front facing (I'm assuming you means transparent, rather)...

Amazon changed how they handled files a week ago.  

As for the deadlock issue, that was several and not actually affecting everyone. 

 

And we've been in "constant contact" with Amazon, meaning we've emailed them every step of the way with all the issues we've had, and we've not received a single reply. That severely limits what we can do... especially as we were hoping they'd respond to us and help us out with the issues we were seeing. 

[CarstenP]

Any hope of getting the increased chunk size anyway so when you get production status we won't get hit by the rampup penalty amazon has on it's services?

[Christopher (Drashna)]

https://stablebit.com/Admin/IssueAnalysis/20799

 

Since I can't seem to find an issue for this already, I've re-flagged the request:

https://stablebit.com/Admin/IssueAnalysis/20799