Ridiculous questions on CloudDrive

[Penryn]

1. If I create a CloudDrive using the local disk provider, is there any sort of chunk checksumming that takes place?
1a. Does the chunk verification work on local drives?
1b. Does chunk verification apply any ECC?

2. Is it bad practice to store a local disk CloudDrive on a DrivePool?

3. Is it bad practice (aside from being slow) to store a local disk CloudDrive on a DrivePool that has a Google Drive CloudDrive as part of the DrivePool drives? ie: a CloudDrive that relies on DrivePool which relies on another CloudDrive.

[Penryn]

Update: I tried using the local disk provider stored on a DrivePool. I transferred a 300MB file over and then it crashed, as in ALL drives appeared disconnected from the DrivePool and CloudDrive, and no drives--not even the boot SSD--appeared in the Task Manager. Upon restarting, neither the DrivePool nor CloudDrive service started successfully and I cannot access Disk Management.

One restart later, after uninstalling CloudDrive everything seems OK.

PSA: CLOUD DRIVE ON DRIVEPOOL: DO NOT TRY THIS! Edit: Tracking available via https://stablebit.com/Admin/IssueAnalysis/27859

[Alex]

On 6/20/2018 at 5:17 PM, Penryn said:

1. If I create a CloudDrive using the local disk provider, is there any sort of chunk checksumming that takes place?
1a. Does the chunk verification work on local drives?
1b. Does chunk verification apply any ECC?

2. Is it bad practice to store a local disk CloudDrive on a DrivePool?

3. Is it bad practice (aside from being slow) to store a local disk CloudDrive on a DrivePool that has a Google Drive CloudDrive as part of the DrivePool drives? ie: a CloudDrive that relies on DrivePool which relies on another CloudDrive.

1. By default, no. But you can easily enable checksumming by checking Verify chunk integrity in the Create a New Cloud Drive window (under Advanced Settings).

1a. For unencrypted drives, chunk verification works by applying a checksum at the end of every 1 MB block of data inside every chunk. When that data is downloaded, the checksum is verified to make sure that the data was not corrupted. If the drive is encrypted, an HMAC is used in order to ensure that the data has not been modified maliciously (the exact algorithm is described in the manual here https://stablebit.com/Support/CloudDrive/Manual?Section=Encryption#Chunk Integrity Verification).

1b. No, ECC correction is not supported / applied by StableBit CloudDrive.

2. It's perfectly fine. The chunk data can be safely stored on a StableBit DrivePool pool.

3. Yep, you should absolutely be able to do that, however, it looks like you've found a bug here with your particular setup. But no worries, I'm already working on a fix.

One issue has to do with how cloud drives are unmounted when you reboot or shutdown your computer. Right now, there is no notion of drive unmount priority. But because of the way you set up your cloud drives (cloud drive -> pool -> cloud drive), we need to make sure that the root cloud drive gets unmounted first before unmounting the cloud drive that is part of the pool. So essentially, each cloud drive needs to have an unmount priority, and that priority needs to be based on how those drives are organized in the overall storage hierarchy.

Like I said, I'm already working on a fix and that should be ready within a day or so. It'll take a few days after that to run the driver through the Microsoft approval process. A fix should be ready sometime next week.

Bug report: https://stablebit.com/Admin/IssueAnalysis/27859

Watch the bug report for progress updates and you can always download the latest development builds here:
http://wiki.covecube.com/Downloads
http://dl.covecube.com/CloudDriveWindows/beta/download/

As far as being ridiculous, I don't think so  I think what you want to do should be possible. In one setup, I use a cloud drive to power a Hyper-V VM server, storing the cache on a 500GB SSD and backing that with a 30+ TB hybrid (cloud / local) pool of spinning drives using the Local Disk provider.

[Penryn]

12 hours ago, Alex said:

1. By default, no. But you can easily enable checksumming by checking Verify chunk integrity in the Create a New Cloud Drive window (under Advanced Settings).

1a. For unencrypted drives, chunk verification works by applying a checksum at the end of every 1 MB block of data inside every chunk. When that data is downloaded, the checksum is verified to make sure that the data was not corrupted. If the drive is encrypted, an HMAC is used in order to ensure that the data has not been modified maliciously (the exact algorithm is described in the manual here https://stablebit.com/Support/CloudDrive/Manual?Section=Encryption#Chunk Integrity Verification).

1b. No, ECC correction is not supported / applied by StableBit CloudDrive.

2. It's perfectly fine. The chunk data can be safely stored on a StableBit DrivePool pool.

3. Yep, you should absolutely be able to do that, however, it looks like you've found a bug here with your particular setup. But no worries, I'm already working on a fix.

One issue has to do with how cloud drives are unmounted when you reboot or shutdown your computer. Right now, there is no notion of drive unmount priority. But because of the way you set up your cloud drives (cloud drive -> pool -> cloud drive), we need to make sure that the root cloud drive gets unmounted first before unmounting the cloud drive that is part of the pool. So essentially, each cloud drive needs to have an unmount priority, and that priority needs to be based on how those drives are organized in the overall storage hierarchy.

Like I said, I'm already working on a fix and that should be ready within a day or so. It'll take a few days after that to run the driver through the Microsoft approval process. A fix should be ready sometime next week.

Bug report: https://stablebit.com/Admin/IssueAnalysis/27859

Watch the bug report for progress updates and you can always download the latest development builds here:
http://wiki.covecube.com/Downloads
http://dl.covecube.com/CloudDriveWindows/beta/download/

As far as being ridiculous, I don't think so  I think what you want to do should be possible. In one setup, I use a cloud drive to power a Hyper-V VM server, storing the cache on a 500GB SSD and backing that with a 30+ TB hybrid (cloud / local) pool of spinning drives using the Local Disk provider.

Thanks for the answers and support, Alex. I understand now that the integrity verification is for revealing corruption/tampering at the source but correction is a non-goal. 

One more for you then:

1c. If a CloudDrive fails verification in a DrivePool, does DrivePool treat the drive as corrupted and pull data from an available duplicated source?

Your explanation makes sense. I was wondering how nested dependencies were supposed to work during loading. I also saw the unmounting service and I think you answered why it exists.

I have another 3 weeks left in my CloudDrive trial, so hopefully I can try the fix out. My college recently gave all alumni "unlimited" GDrive space and I'd like to use whatever my network can muster. Unfortunately during the whole CloudDrive on DrivePool fiasco I lost a drive to the frequent power cycling and hard shutdowns. It now only initializes properly at an angle (???).

That Hyper-V setup sounds just as ridiculous--it's assuring to know I'm in good company. Thanks again for all your (and Chris's) hard work! DrivePool and Scanner have been great to me over the years.

[Alex]

1c. If a cloud drive fails the checksum / HMAC verification, the error is returned to the OS in the kernel (and eventually the app caller). It's treated exactly the same as a physical drive failing a checksum error. StableBit DrivePool won't automatically pull the data from a duplicate, but you have a few options here.

First, you will see error notifications in the StableBit CloudDrive UI, and at this point you can either:

• Ignore the the verification failure and pull in the corrupted data to continue using your drive. In StableBit CloudDrive, see Options -> Troubleshooting -> Chunks -> Ignore chunk verification. This may be a good idea if you don't care about that particular file. You can simply delete it and continue normally.
• Remove the affected drive from the pool and StableBit DrivePool will automatically reduplicated the data onto another known good drive.

And thank you for your support, we really appreciate it.

As far as Issue #27859, I've got a potential fix that looks good. It's currently being qualified for the Microsoft Server 2016 certification, Normally this takes about 12 hours, so a download should be ready by Monday (could even be as soon as tomorrow if all goes well).