Jump to content
Covecube Inc.
  • 0
modplan

Full Cache Re-upload after crash?

Question

Sorry if this has been covered a quick search did not find me what I was looking for. 

 

I have a CloudDrive that I send windows server backups to nightly. The full backup size is about 75 GB, but the nightly incremental change is only 6-7GB, easily uploadable in my backup window. 

 

I have set the cache on this drive to 100GB to ensure the majority of the "full backup" data is stored in cache, so that when windows server backup is comparing blocks to determine the incremental changes, clouddrive does not have to (slowly) download ~75GB of data for comparison every single night. 

 

This works very well.

 

The problem comes when there is a power outage, crash, BSOD, etc. Even though the clouddrive is fully current and "To Upload" is 0, when I bring the server back up, after we go though drive recovery (which takes 5-8 hours for this 100GB), cloud drive then moves ALL 100GB of the cache into "To Upload" and starts re-uploading all of that data.

 

Why? I can't think of how this is necessary. Incase a little data was written to the cache at the last minute before the unexpected reboot? If so, certainly there is a better way of handling this than a 100GB re-upload, some sort of new-unuploaded-blocks tag/database? What if a drive has a massive cache, a re-upload could take days/weeks!

 

Thanks for any insight, I've gone through this process a couple of times using clouddrive and it has been painful every time. I'd be happy even if we downloaded every single block that is cached, compared it to the local cache block, and then only uploaded the ones that have changed.

Share this post


Link to post
Share on other sites

19 answers to this question

Recommended Posts

  • 0

What OS are you using?

What version of StableBit CloudDrive are you using?

 

What is the size of the drive?

What is the size of the cache (You've indicated that it's 100GBs, but I'm not entirely sure about this). 

 

 

To clarify, did you restore any files or the system from the backup, and then continue backing up the system? 
If so, that *may* have been the cause of the issue here, specifically.

 

Otherwise, could you let me know exactly what happened?

Share this post


Link to post
Share on other sites
  • 0

What OS are you using?

What version of StableBit CloudDrive are you using?

 

What is the size of the drive?

What is the size of the cache (You've indicated that it's 100GBs, but I'm not entirely sure about this). 

 

 

To clarify, did you restore any files or the system from the backup, and then continue backing up the system? 

If so, that *may* have been the cause of the issue here, specifically.

 

Otherwise, could you let me know exactly what happened?

 

Sorry, info I know I should have provided. 

 

2k8 R2

.470 (But have seen this several times while using CloudDrive, any time there is an unexpected reboot basically, version does not matter)

Drive is 10TB 

Cache is currently 100GB, I've been playing with different sizes.

 

No files have been restored. In fact this seems to have nothing to do with running a backup, in my experience, any local cache data is moved to "To Upload" after "Drive Recovery" after an unexpected reboot. This is just the only drive I currently have a Cache on. I have seen this on other drives just with standard files, when I had a cache on them.

 

 

Basically:

- Create Drive with a cache

- Add data to the drive, ensure that the cache has some data in it

- Ensure "To Upload" = 0B

- Pull the power plug/force BSOD

- On Boot, drive will go through lengthy "Drive Recovery" procedure

- After "Drive Recovery", ALL local data from the cache is added to "To Upload" and re-upload begins, ALL this data already exists in the cloud

- If you have a large cache, this takes forever

 

Hope that makes sense?

Share this post


Link to post
Share on other sites
  • 0

http://community.covecube.com/index.php?/topic/1610-how-the-stablebit-clouddrive-cache-works/

 

Might be by design:

 

 

So what happens when you pull the plug in the middle of this process?
  • StableBit CloudDrive loses  the "To Upload" state. Well, it doesn't really lose it, but it's in an indeterminate state, and we can't trust it any longer.
  • In order to recover from this, StableBit CloudDrive assumes that all locally cached data was not uploaded to the provider.
  • It is safe to make this assumption because uploading something that has already been uploaded before doesn't corrupt the drive as a whole in the cloud. While, not uploading something that needed to be uploaded would be catastrophic. Because that would mean that your data in the cloud would get "out of sync" with your local cloud drive cache and would get corrupted.

 

  •  

Share this post


Link to post
Share on other sites
  • 0

 

Thanks thnz, looks like you are right. 

 

Which leads to a broader conversation. Is this the right way to do this for drives with large caches? Could a "chunk tracking database" that marks local cached chunks as perfectly uploaded be used to prevent the wholesale re-upload of the cache? Only re-upload the chunks that were not marked in the database as previously successfully uploaded? If someone with somewhat limited bandwidth sets a 500GB cache on a large drive, suffers a power outage, but 498GB of that cache was previously perfectly uploaded, this wholesale re-upload would take weeks.

 

Edit: If a "chunk tracking database" is not viable, maybe download the chunks and compare the checksums to the local chunks? Most people have MUCH faster download, than upload. So only the needed chunks would be re-uploaded.

Share this post


Link to post
Share on other sites
  • 0

Experienced this strange behaviours already as well. The funny thing is that after the crash some drives that were fully uploaded after recovery were just fine - while others were re-uploadiung everything.....

 

Might be by design:

 

So it sounds like a poor design. Can't imagine any database doing this after unclean shut down.

Apart from that - if I will loose the drive where local cache was stored - assuming everything was completely uploaded - doest that mean I will loose my data stored in DriveCloud?

 

For me that means that the cache should be as little as possible / as little as you can upload in a accepteble period of time.

 

With this or you';ll "force" the app to trust what is uploaded or ... loose your data.

Share this post


Link to post
Share on other sites
  • 0

I've noticed a few times the drive will go into recovery even after a clean computer restart, needing to re-upload the entire cache. I get the 'unsafe shutdown' message even though Windows restarted gracefully.

Share this post


Link to post
Share on other sites
  • 0
thnz, on 30 Apr 2016 - 4:36 PM, said:

I've noticed a few times the drive will go into recovery even after a clean computer restart, needing to re-upload the entire cache. I get the 'unsafe shutdown' message even though Windows restarted gracefully.

I've had this exact same issue on Windows Server 2012R2.  I perform a normal Windows reboot and I have to re-upload everything...  It's very annoying.  I really like the Cloud Drive I just need this fixed.  I've got 300 down but only 30 up. 

 

Please do something to fix this!

Share this post


Link to post
Share on other sites
  • 0

Experienced this strange behaviours already as well. The funny thing is that after the crash some drives that were fully uploaded after recovery were just fine - while others were re-uploadiung everything.....

 

 

So it sounds like a poor design. Can't imagine any database doing this after unclean shut down.

Apart from that - if I will loose the drive where local cache was stored - assuming everything was completely uploaded - doest that mean I will loose my data stored in DriveCloud?

 

 

With this or you';ll "force" the app to trust what is uploaded or ... loose your data.

 

It isn't poor design though. If it shuts down randomly in the middle of an upload. How will it know what it has uploaded yet or not. The data is unreliable it is better to reupload the entire cache than to deal with the data inconsistencies. I personally clear cache often, this prevents that from happening to me. It has happened and it is annoying however it is one of those things that makes sense. And if you attempt a shut down and you force the shut down on windows. That will kill processes that are still active and give you a problem when you restart it.

Share this post


Link to post
Share on other sites
  • 0

I see the latest beta version (.818) contains the following change:

 

 

* [D] Cloud drives were sometimes not shutting down properly, unnecessarily forcing unsafe recovery.

 

Hopefully that fixes this issue, and I can now go back to a much larger cache size without the risk of it going into recovery so often after a restart.

Share this post


Link to post
Share on other sites
  • 0

I see the latest beta version (.818) contains the following change:

 

 

Hopefully that fixes this issue, and I can now go back to a much larger cache size without the risk of it going into recovery so often after a restart.

 

This should EXPLICITLY fix the issue.

 

For whatever reason, there was a bug introduced that was allowing the service to be shut down before it was done running it's "shutdown routines".  This was causing the data to end up in the "unsafe" state after rebooting/shutting down. 

 

This fix corrects that issue and should return to working properly. 

 

 

 

Ironically, this seems to have effected drives with larger caches more. So the larger the cache was, the more likely this was to occur, and the more data it would have to verify.  :(

Share this post


Link to post
Share on other sites
  • 0

Windows 10 x64, and CloudDrive beta 850.

 

Looking over the system event logs, it did mention an unexpected shutdown when It was supposedly shut down safely, so there may well be something else going on preventing a clean shut down. I'll have a closer look later on.

Share this post


Link to post
Share on other sites
  • 0

Did StableBit CloudDrive report the unsafe shutdown, or did Windows (eg, the event logs). 

 

If Windows did, then the system may actually be crashing when shutting down,  Running a utility like "BlueScreenViewer" may be a good idea, as it may pick up the crash dump from what the shutdown occurred. 

Share this post


Link to post
Share on other sites
  • 0

Did StableBit CloudDrive report the unsafe shutdown, or did Windows (eg, the event logs). 

 

If Windows did, then the system may actually be crashing when shutting down,  Running a utility like "BlueScreenViewer" may be a good idea, as it may pick up the crash dump from what the shutdown occurred. 

 

It was reported both in CloudDrive and in the system event logs, so I'm now assuming the system was actually crashing while shutting down, and CloudDrive was doing what it was supposed to by recovering. Unfortunately CCleaner has since removed any dump files, so I'm not able to investigate any further at the moment. 

 

As tends to happen with these things, I've been unable to reproduce it since. I've excluded system memory dumps from CCleaner in the future, so if/when it happens in the future, I should have more to work with - it could well just be a wobbly driver or something.

Share this post


Link to post
Share on other sites
  • 0

If that is the case, then yea, it sounds like it was crashing/BSODing during shutdown, which is ... tricky at best. 

 

And ... yeah, the quantum tech support issue.  If you're watching for the issue, it won't happen.  Seriously.  I've ... experienced that more than a few times.... and have had a few remote support sessions like that.  It's frustrating. 

 

 

Though, maybe CCleaner cleared out whatever was causing it...  

But if it does come back, please do grab those crash dumps!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...