Upload Verification vs Just a checksum check

[WBadminUser]

I absolutely love this product just which i had a better back-end then Amazon cloud drive.

My question is about the option of Upload Verification. This actually pulls the block down to verify it? with this option not checked i assume it still does a checksum check on the data. 

 

Thanks for the clarification.

 

 

[WBadminUser]

Never mind i found the tool tips that explain it, and it works like i thought it did. 

That checkbox to ignore checksums is pretty scary though.

[Christopher (Drashna)]

Never mind i found the tool tips that explain it, and it works like i thought it did. 

That checkbox to ignore checksums is pretty scary though.

Generally, it's not needed though. Most providers are very good about getting the data properly uploaded, or erroring out if there was any issues.

 

However, some providers (such as Amazon Cloud Drive, especially) do run into errors. This is to ensure that they're uploading the data correctly *and* that it's actually there.

 

 

That said, yes, it does pull the data back down once it's uploaded, to make sure that it's been properly uploaded and that the blocks of data are intact.  Once it has, it moves onto the next chunk... otherwise, it re-uploads the file.

[Alex]

I absolutely love this product just which i had a better back-end then Amazon cloud drive.

My question is about the option of Upload Verification. This actually pulls the block down to verify it? with this option not checked i assume it still does a checksum check on the data. 

 

Thanks for the clarification.

 

StableBit CloudDrive performs a checksum check after downloading every chunk, if and only if, the Use checksum verification checkbox is checked when you first create the drive.

It's always enabled by default for cloud providers, and not available as an option for the Local Disk and File Share providers.

 

The difference between using upload verification vs. simply just checksum verification is that, checksum verification is performed much later. That is, it's performed when you need to download that chunk from the provider as a result of a local read request from the local cloud drive.

 

While upload verification is performed at upload time. It uploads a chunk of data to the cloud and then downloads it to make sure that what was downloaded is the same data that was uploaded. And here's the key difference, if it's not, then that chunk upload will be retried. If multiple retries fail, you will be notified in the UI and uploading will halt for 1 minute after which more retries will follow.

 

Upload verification protects your data from the scenario where the cloud provider receives your data and then saves it incorrectly, or if the data was corrupted in transit. Upload verification will detect this scenario and correct it, while checksum verification will detect it much later, and not correct it.

 

How common is this? From my testing, and remember this is still a 1.0 BETA, I've already seen this happen. And specifically I've seen it happen with Amazon Cloud Drive, where an uploaded file would not download at all. I've confirmed this with the Amazon Cloud Drive web interface, so it definitely was not a bug in our software. The file got into some kind of corrupted state where its metadata was there, but its contents were gone.

 

So for non enterprise-grade providers, I really recommend that upload verification be enabled.

 

What is the point of disabling checksum verification at download time?

 

If you run into the scenario where your data is saved incorrectly to the cloud, and you didn't have upload verification enabled. When you try to read that data back, you will get a checksum verification error. This is exactly how a physical hard drive works, and this is similar to having a "bad sector". Your hard drive will never allow corrupt data to be read back, it will retry over and over again in hopes that the original data can be recovered. At this point your system may lock up or freeze temporarily because it can't read or write to the drive in a timely manner.

 

This is the exact behavior that StableBit CloudDrive will emulate, except that the data being read back is now coming from the cloud. If for some reason you want to break out of this "loop", and allow the corrupt cloud data to be read back, you can toggle Ignore checkum on, and that corrupt data will be read back.