Does CloudDrive cause these bluescreens?

[worldtravelgeek]

Hi !

 

I have been using CloudDrive on my Windows 10 notebook (version 1607) for a few days now and since that time I have experienced three bluescreens in Windows already. They always happen, after a Windows start, right after the lockscreen appears. Upon the reboot everything is fine. I have checked the dump file Windows created with Bluescreen View but I couldnt find any reference to CloudDrive. However, I never had this problem before using CloudDrive so ...

 

One other thing to notice is, that this only seems to happen when CloudDrive cannot connect to the internet after a Windows start, since I am traveling atm and connecting to various wifi. At home I didnt have the crashes, maybe because there I have a constant connection. No idea if thats related though.

 

Here is a picture of Bluescreen View and the Cloud Drive log after a restart. If you need anything else, let me know. I am sure the crash will happen again tomorrow ;( Unfortunately, CloudDrive always has to perform a recovery of the drive after such a crash and apparently keeps uploading the same 10 GB of data again and again and again after this (

 

0:00:29.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:30.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:31.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:32.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:33.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:34.3: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:35.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:36.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:37.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:38.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:39.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:40.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:41.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:42.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:43.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:44.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:45.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:46.4: Warning: 0 : [CloudDrives] Not connected to the Internet. An exception occurred during a Ping request.
0:00:47.5: Information: 0 : [CloudDrives] Synchronizing cloud drives...
0:00:47.6: Information: 0 : [CloudDrives] Synchronizing cloud drives...
0:00:47.6: Information: 0 : [CloudDrives] Valid encryption key specified for cloud part 0780ec92-ac48-430f-98c1-2898b975c7f9.
0:00:47.6: Information: 0 : [Main] Enumerating disks...
0:00:48.0: Warning: 0 : [ValidateLogins] Error instantiating management provider. The remote name could not be resolved: 'login.live.com'
0:00:48.0: Warning: 0 : [CloudDrives] Cannot start I/O manager for cloud part 0780ec92-ac48-430f-98c1-2898b975c7f9. The remote name could not be resolved: 'login.live.com'
0:00:48.3: Information: 0 : [CloudDrives] Synchronizing cloud drives...
0:00:48.3: Warning: 0 : [CloudDrives] Cannot start I/O manager for cloud part 0780ec92-ac48-430f-98c1-2898b975c7f9. The remote name could not be resolved: 'login.live.com'
0:00:49.0: Information: 0 : [Disks] Updating disks / volumes...
0:00:49.1: Information: 0 : [Main] Starting disk metadata...
0:00:49.1: Information: 0 : [Main] Updating free space...
0:00:49.1: Information: 0 : [Main] Service started.

[Christopher (Drashna)]

Normally, BSODs caused by StableBit CloudDrive would be caused by "cloud_fs.sys", as that's out driver. 

 

however, since we are a kernel level driver, it could be an interaction in the kernel that is causing the issue. 

 

 

What version of StableBit CloudDrive are you using? 

Additionally, if you disable the StableBit CloudDrive Service (boot into safe mode, run "services.msc", find the service, right click on it and select "Properties"... then set the startup type to "manual" or "disabled"), does this fix the issue? 

 

Better yet, could you try this version: 

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.696_x64_BETA.exe

 

If that doesn't help, then could you upload the crash dumps? 

http://wiki.covecube.com/StableBit_CloudDrive_System_Crashes

[worldtravelgeek]

Hi!

 

 

 

What version of StableBit CloudDrive are you using? 

 

I was using the 687 Beta. I will update to the version you mentioned and report back

[worldtravelgeek]

Hi again,

 

ok, the new version unfortunately didnt help. Plus I did as you instructed and disabled the services. I then tried three shut downs and no more crash appeared. I then enabled the services again and upon the first reboot another crash. However this time not on the lockscreen but only once I started the Cloud Drive UI. So I guess, that kinda proves the case. I am uploading the dump file now. 

 

Oh an btw ... why does CD always have to upload the same 10 GB of data after each crash? Shouldnt it delete files from the "To Upload" queue once they are verified on the cloud drive? Why keep uploading the same stuff over and over and offer? 10 GB is a large amount of data, I would understand if it was a few MB of the last uploaded chunks but not everything it uploaded in a few days ?!?!

[Christopher (Drashna)]

Thanks for uploading the dump. 

 

And yes, this is definitely StableBit CloudDrive related. 

I've flagged the issue for Alex (the developer) so he can take a look.

https://stablebit.com/Admin/IssueAnalysis/27153

 

 

As for the 10GB that gets reuploaded, this is a check/verification of the cache, IIRC. When unsafe shutdowns occur (eg BSODs), we can't trust the cache until we can verify it.

However, if you're sure it's the same 10GB (click on the gear, select the technical details).  then let me know.

[worldtravelgeek]

I have to say I have deactivated CloudDrive on my notebook by now. The crashes have intensified lately and combined with the reuploading of the same data, I the whole cloud drive is pretty useless for me anyway ...

[Christopher (Drashna)]

For the time being, yeah, I think that's best.  

 

Or set the service startup type to "manual", so that the drives are only mounted when you want. (eg, when you start the service)

[Christopher (Drashna)]

By chance, was the cache drive on a removable (USB) drive? 

 

If so, that appears to be the cause of the issue here.  USB drives are "allowed" by specification to unmount unexpectedly. This causes serious issues for the cache. 

 

Unfortunately, the "fix" is to essentially block the ability to place the cache on a USB drive.  This may break functionality for your setup, as it will essentially require you to use an internal disk. And in your case, that sounds like the system disk, only. 

[worldtravelgeek]

Sorry for the late reply. Havent had any time to check the forum lately.

No, the cache drive was NOT on a removable drive. It was on a second SSD in my notebook (I added a second SSD drive) but it is a fixed M2 style drive, not removable. 

[Christopher (Drashna)]

Not a problem. 

 

The 720-721 build is the newest, and if that doesn't fix the BSOD's, let us know. 

 

 

But as for removable, even if it's SATA, M.2, NVMe, etc, it may still show up as a removable drive. Some intel controllers are bad about this (older ones), and some AMD chipsets are still bad about this.  If the drive shows up in the "safely remove hardware" option, it's being counted as "removeable". 

[worldtravelgeek]

Hi Christopher,

 

no, the drive does not show up in the "safely remove hardware" option. It is not removable So I doubt this was the cause of the problem. Maybe it is something else, for instance the fact that the drive is bitlocker protected or not the system disk. Anyway, I will test the new build, even though I am back home now for a few weeks, where I have stable wifi and the problem seemed to be related to unstable/not available internet connection upon restart. 

[worldtravelgeek]

Ok,

 

problem is NOT solved. Just got a crash again with the 721 beta version. And again, it happened while I had no or unstable internet access during bootup. I think THIS is the key to the problem, not the removable drive thing you mentioned, which was not even removable in my case.

 

I am uploading the new crash dump while I am writing this. I had an short internet outage at home, meaning the wifi connection was still on but there was no internet connection. After I saw this, I took my chance and rebooted the notebook where Clouddrive was running and uploading data. Right after the lock screen appeared, the crash happened. Another reboot later, I was able to log in and saw that Cloud Drive had to reupload the last 20 GB of data or so.

 

Really, I am kinda disppointed by my Cloud drive experience up until now. I have to say that Sorry

I know its still beta but I did think it would be a bit more stable after so many versions

 

Oh well. My whole experience with bad internet while traveling made me think about my initial plans to upload everything to the cloud anyway. It doesnt seem workable unless I always have a high speed connection which most of the time is not the case. Maybe it is better to invest in a small external 2 TB SSD (or multiple ones)

 

PS: 
I cannot upload the 150GB dump file. (it is already zipped)
Is there enough storage left on your dropbox? I tried at least 20 times now but always breaks up at 4%. (my internet seems to work now though)

Edited September 6, 2016 by worldtravelgeek

[Christopher (Drashna)]

problem is NOT solved. Just got a crash again with the 721 beta version. And again, it happened while I had no or unstable internet access during bootup. I think THIS is the key to the problem, not the removable drive thing you mentioned, which was not even removable in my case.

 

I am uploading the new crash dump while I am writing this. I had an short internet outage at home, meaning the wifi connection was still on but there was no internet connection. After I saw this, I took my chance and rebooted the notebook where Clouddrive was running and uploading data. Right after the lock screen appeared, the crash happened. Another reboot later, I was able to log in and saw that Cloud Drive had to reupload the last 20 GB of data or so.

 

Okay, thank you for the clarification/testing, as that should help immensely 

 

And just in case, could you grab boot time logs?

http://wiki.covecube.com/StableBit_CloudDrive_Boot_Time_Log_Collection

 

 

Really, I am kinda disppointed by my Cloud drive experience up until now. I have to say that Sorry

I know its still beta but I did think it would be a bit more stable after so many versions

 

Oh well. My whole experience with bad internet while traveling made me think about my initial plans to upload everything to the cloud anyway. It doesnt seem workable unless I always have a high speed connection which most of the time is not the case. Maybe it is better to invest in a small external 2 TB SSD (or multiple ones)

 

Well, it's a really complex piece of code, with a lot of moving parts.  But still.  

 

However, in most cases, it is pretty much rock solid, at this point. At least depending on the provider in question 

 

However, flaky internet will definitely cause issues. Even if that is just filling up the cache drive. 

 

 

PS: 

I cannot upload the 150GB dump file. (it is already zipped)

Is there enough storage left on your dropbox? I tried at least 20 times now but always breaks up at 4%. (my internet seems to work now though)

 

I'm assuming that's a 150MB dump file. I mean, unless you have 150+GB of RAM installed on your system. And if so, kudos.

 

Should have plenty of space on the DropBox account.  IIRC, it's a TB (or more) and would take a lot more to fill it. 

 

I've also tested a 120MB file (actually, one of yours), and it uploaded file.  So it may be a flaky connection that is the issue, ironically(?) that is the problem. 

If that's the case, feel free to break it up into parts, or uploading it somewhere else and give me a link. 

[worldtravelgeek]

Hi Christopher,

 

ok. I hope we active betatesters with lots of crash experience at least get a free license after the beta period ended  

So, I am uploading the memory dump and the service logs with boot time logging here:

 

https://www.dropbox.com/sh/fhsol488hg4ot2v/AAD-IY0ernZaV_J7Yl79mP5Pa?dl=0

Usually my internet connection at home is pretty stable, not sure why I cannot upload with your dropbox uploader. It always breaks up at exactly 4-5% but uploaded fine to my dropbox

 

Oh and yes, it is MB, not GB of course. I wish I had that much RAM  

Not sure, thats worth mentioning btw but I had to stop the service to be able to zip the log files after a restart, or 7zip would complain about not being able to access the data.

 

As for flaky internet causing problems:
 

It shouldnt be that flaky imo.   It shouldnt cause crashes first of all. Second, I dont understand why CD has to reupload 10-20 GB of data every time the drive is in recovery. Shouldnt it verify a file/chunk after upload and then mark it as verified and remove it from cache? Instead CD seems to upload GBs and GBs of data without verifying it and then when a crash occurs, it has to reupload the same GBs of data. This does not make any sense. I understand, you need to trust the uploaded data. But please make the necessary reupload as small as necessary, not as big as possible 

[Christopher (Drashna)]

Okay, I've downloaded and uploaded the files.  And honestly, I have NO idea why you can't upload them. But I'd place money on it being related to the CloudDrive issues.

 

I've flagged the files for Alex:

https://stablebit.com/Admin/IssueAnalysis/27176

 

 

 

As for GB, I've seen people with that.

You can find cheap RAM online. But you need a good, server board to host that much. (by HyperV system, the board supports up to 192GB, but I'm not using anywhere near that much)

 

 

Hopefully, Alex can identity what's going on and fix the issue. 

[worldtravelgeek]

Did you guys ever find a solution for this problem?

[Christopher (Drashna)]

Not yet. Sorry. 

 

I'll bump the issue. 

[worldtravelgeek]

Hi !

 

Any idea, when there might be a solution to this? I would have thought, that a bluescreen issue like that would be kinda high priority.   
I am still experiencing them occasionally btw. so last time it happened, I didnt ever bother to mount the drive anymore. 
Anyway, apart from that issue I am very unhappy with using the cloud drive on the road anyway. I dont really know how everyone is so happy about using cloud drive. I am having nothing but problems with it. Not only the constant crashing issues but even if it works, it is so extremely painfully slow to use the data on the cloud drive that it is actually not possible to use it. Like if I want to listen to mp3 files on the drive, I get constant skips and errors every few seconds. Granted, I am not always on great internet speeds in hotel wifi hotspots but thats what cloud drives are for ... to access my data from anywhere. If it only worked well at home with my 100mbit connection I might as well use local storage. Oh well ...

[Christopher (Drashna)]

I know this is a shit excuse, but Alex does have a lot on his plate right now.  Between the Google File ID issues, DrivePool measurement code, a really weird VSS bug with CloudDrive, he's been very busy, 

 

However, I've bumped this. 

[Christopher (Drashna)]

Okay, Alex has taken a look at this. 

 

Unfortunately, it looks like the drive being used for the cache (cache drive) is being unmounted suddenly. Eg, it's being pulled, the controller is dropping it, etc. 

The problem is that there isn't a good way to handle this. Or rather, the ONLY good way to handle this ... is how we are: a crash (BSOD).  

 

 

Alex has posted detail about the crash and what is going on here. 

http://stablebit.com/Admin/IssueAnalysis?Id=27176

 

The best option may be to use a different drive, or use a drive on a different controller. 

 

Speaking of which, what hardware are you using here? And how is the drive in question connected to the system?

[worldtravelgeek]

Hi Christopher and Alex,

 

first, thanks for taking a look into this. However, this is crazy. The hardware is an Acer R13 R7-372T notebook with two internal M2 SSDs with Intel RST drivers installed. This is a relatively new notebook (from this year) and absolutely stable without Clouddrive installed. Just yesterday, I copied about 200 GB of data to the second drive (the one that CD used as the cache drive previously) and did not get any bluescreen.

 

What you are saying basically is, that the drive disconnects while in use. This would imply that it is a faulty drive but then I would see the same bluescreens during general use, which I don't. In fact, even if CD is installed but no CD is mounted, there are no Bluescreens. 

 

Second, like I mentioned earlier, most of the time I get those bluescreens right after a restart when the Windows lockscreen appears and if there is no or no stable WiFi connection available. From experience, I would say, that the bluescreen occurs, if CD tries to connect to the drive but cannot or not reliably. But I doubt that is because of a disconnecting local drive, I would say it is because of a network related issue and it seems weird that this cannot be handled in a different way.

 

Anyway, I am not an engineer so I cannot argue on a deeper technical level, the explanation just does not seem plausible from my experience.

[Christopher (Drashna)]

To clarify, the BSOD is being caused BY the drive being dismounted.  Period.  There is absolutely no question of that. The crash dump *clearly* indicates that this is what is happening. 

 

That said, if this is not happening consistently, then it may be other moving parts here, such as the controller getting overworked, or a specific set of instructions to the controller that, or simply a timing issue that is triggering this.  

 

 

What Alex is seeing in the dump is a "KERNEL_DATA_INPAGE_ERROR" that is occurring in the read map. 

This is occurring because the drive basically isn't present. 

 

As Alex illustrates, the same would happen (in theory) if the page file existed on the disk, as it's essentially the same issue. 

 

 

 

Regardless, if this is ONLY happening during startup, then try setting the service startup type for the "StableBit CloudDrive Service" (NOT the "StableBit CloudDrive Native Service") to "Automatic (Delayed)".  This will delay the loading of StableBit CloudDrive by approximately two minutes, giving the rest of the system time to start. 

 

To do this, run "services.msc", find the server, right click on it and select "Properties". Find the startup type (roughly half way down the window) and change that from "Automatic" to "Automatic (Delayed)" and reboot the system. 

 

If the issue continues to occur at the same point, then let me know. 

 

Another option may be to uninstall or reinstall the Intel RST drivers (or update them).  We've seen more than a few ODD issues being caused by these drivers. 

Unfortuantely, I suspect that the M.2 support requires the drivers, so upgrading (the drivers) may be the best option here. 

 

 

Anyway, I am not an engineer so I cannot argue on a deeper technical level, the explanation just does not seem plausible from my experience.

 

And that's the thing, crash dumps are a snapshot of the memory at the time of the error.  They're like a snapshot of a crime scene at the exact moment the crime occurs.  These are very detailed and if you know how to read them (and Alex definitely does), you can usually get a very clear idea of what is going on. 

 

And the dumps you've provided, it's very clear what the issue is.  The volume that the cache is on has been dismounted when we try to read from it.  *that* is specifically what is causing the BSOD. 

 

 

Though, that reminds me.... do you get a "Safely remove hardware" entry for the drive used for the cache? 

If so, that may be part of the problem. 

[worldtravelgeek]

Hi Christopher,

 

the question that arises is ... why is my notebook totally stable without Cloud drive? If the drive, the controller or the drivers would cause the drive to disconnect from time to time, I would certainly get bluescreens all the time. But as I said, I just copied 200 GB of date to that drive and watched videos from it. Not a single time, did I get a bluescreen. So why is it only occuring in combination with CD? 

 

Anyway, at the moment I am not sure if I want to pursue this cause it is time consuming. I might change the cache drive to the first drive, I might delay the start of the service and check if it works. I might try to use the Microsoft storage drivers instead of the Intel ones although they are said to cause more battery drain because it does not let the SSDs go into deep sleep apparently. Having said that ... could this be the problem? The driver telling the drive to go into a deeper sleep state while CD tries to read from it? Mhh ....

 

Anyway, I will let you know if I get around to testing any of these options.

Alex

[Christopher (Drashna)]

Don't get me wrong, I'm not trying to say that StableBit CloudDrive isn't involved with these crashes. 

 

What i'm saying is that it's not what is triggering them.  It is trying to read from the disk(s) in question, and at the time the BSOD is triggered, the drive is not present.   It may be a complicated timing issue on the system. 

 

And as for copying data to the drive versus StableBit CloudDrive? The IO operations what we use are very much different from a
"simple" copy operation, and are more similar to the paging file and paging operations. 

 

 

 

As for the drivers, if they are initiating some sort of deep sleep mode, that may actually be part of the issue here then.  When the drivers initialize during the boot up process, it may actually be caused by a system service (or a driver) at about the same time that the StableBit CloudDrive stuff is being initialized.  So it may be triggering this at *exactly* the wrong time. 

 

Setting the service startup delay for StableBit CloudDrive would most likely fix the issue if this is the case.