I/O deadlock?

dragon2611 · June 27, 2015

Is it possible to get into a situation where the I/O seems to deadlock, I have a 2012r2 machine I was copying 300GB or so of data to onedrive Business including lots of small files,

It Looks like the copying has stopped but also I cannot bring up task manager and a shutdown -r -f -t 30 command is hanging. Hoping that it's cloud drive and not one of the disks in the box is failing.

Christopher (Drashna) · August 12, 2015

I'll install this build on the machine I was trying to use last time and see if it solves the issue.

Edit: Installer sat for a very long time and eventually stated the service failed to start, X64 version on server 2012r2

from the event log.

Service cannot be started. System.TypeInitializationException: The type initializer for 'CloudDriveService.Main' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Cove.Util, Version=1.0.5689.32472, Culture=neutral, PublicKeyToken=4c28787c0620f424' or one of its dependencies. The system cannot find the file specified.
   at CloudDriveService.Main..cctor()
   --- End of inner exception stack trace ---
   at CloudDriveService.Main.#vt()
   at #eae.#Efc.#Bfc(String[] #qRc)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

Was this a clean installation or an upgrade?

And if was an upgrade, to what version and from which version?

Just had the 'deadlock' lockup again. Happened while drivepool was duplicating files to the clouddrive. Will upload a dump shortly.

*edit* Uploaded here. Drive is doing recovery right now.

Okay. Could you uninstall the current version, and make sure that "C:\Program Files\StableBit\CloudDrive" is empty.

If it's not, make sure you manually stop the services for it in "services.msc", use task manager to "end" the notification app, delete the directory, reboot and then reinstall?

dragon2611 · June 27, 2015

Hard rebooted the machine, clouddrive is currently going through recovery/checking.

Christopher (Drashna) · June 27, 2015

If you are able to reproduce this, could you enable logging:

http://wiki.covecube.com/StableBit_CloudDrive_Log_Collection

This should give us a clear idea of what is going on, and help fix the issue.

dragon2611 · June 27, 2015

Presumably I hit the API limits for Onedrive as After reboot I wasn't getting any upload/download.

It actually echos the exeperiance I had with StorageMadeEasy where the onedrive/office365 API was to unstable for handling large files.

thnz · June 27, 2015

Might be the same crash I was having, though I was using Amazon CloudDrive. During heavy I/O (was duplicating via drivepool at the time) the system stops responding (ish). Mouse still moves, still responds to pings, can move windows around (though they're all 'not responding'), media continues to play, but cant ctrl+alt+del, or remote desktop into it - needs a hard restart.

I've been unable to reliably reproduce it so have been unable to get logs/mem dumps when it happens.

As for handling large files - it looks like clouddrive stores data at block level, so individual filesize might not matter.

dragon2611 · June 28, 2015

Same thing I was seeing, I couldn't move interact with explorer or open anything but services that were running on the machine seemed to still be accessable (E.g softether vpn server was still working as I could VPN to the box)

I think the problem is caused by large files or lots of small files, essentially anything that causes a large number of API calls, My best guess is the API gets rate limited or something else causes a disruption to the upload process and when explorer tries to go access the drive it gets stuck waiting for an I/O operation.

Of course I could be way off the mark.

I've uninstalled it for now as I was only really interested in using it with either onedrive or Hubic (not yet supported) and the main reason I was interested was due to the clientside encryption (I.e I want to use these providers but it doesn't mean that I want to have to trust them 100%)

Kevin · June 29, 2015

I think I'm experiencing this as well. Trying to sync ~160 Gb of surveillance data to ACD. After hours my RAM is completely consumed and then at some point the server just reboots. Chiming in to follow the thread.

Christopher (Drashna) · June 29, 2015

@Dragon2611:

OneDrive for Business/Office365 uses sharepoint as the backend, IIRC. And that can become problematic if you have a lot of data. We've implemented workarounds for that but ....

And I do beleive that they throttle the connection.

But yes, files are stored in blocks. Specifically, you can set the "chunk" size, which is the block size that we store the raw data in. Changing these sizes can absolutely change the performance profile for the drive.

Anyone experiencing the lockup, could you get a memory dump?

http://wiki.covecube.com/StableBit_CloudDrive_System_Freeze

I know that StableBit CloudDrive is very memory intensive, but if it's causing issues, that's something that needs to be addressed.

thnz · June 29, 2015

It had another lockup this morning. I've now run that reg tweak and rebooted, so next time it happens I'll hopefully be able to get a dump.

thnz · June 30, 2015

Caught and dumped. Compressing and uploading atm.

This crash occurred during duplication, though it does occur at other times too when the disk isn't part of a drive pool - maybe during I/O of some kind?

thnz · July 3, 2015

Is there any way to see when these issues are looked at? Or should I just keep an eye on the changelog. I'd like to know when it's good to test again of if you need more info/dumps.

Christopher (Drashna) · July 6, 2015

I'm sorry for the delayed response here.

I've flagged the dump for Alex, and he'll take a look at what's going on here.

Issue for this created here:

https://stablebit.com/Admin/IssueAnalysis/17719

thnz · July 10, 2015

It was in Windows 8.1 rather than Server 2012 (not sure if this is important). Could you create another issue for the resize-while-uploading bug too? http://community.covecube.com/index.php?/topic/1297-drivepool-integration-questions/&do=findComment&comment=8697posts #4 and #5.

Christopher (Drashna) · July 14, 2015

Sorry for the delayed response here.

It was in Windows 8.1 rather than Server 2012 (not sure if this is important). Could you create another issue for the resize-while-uploading bug too? http://community.covecube.com/index.php?/topic/1297-drivepool-integration-questions/&do=findComment&comment=8697posts #4 and #5.

The OS can definitely matter, so it's always good to include that info.

As for the resize issue, giving it's timing, and the "deadlock" type issue, I'm not bugging it at this time.

Don't worry, we're not ignoring it. However, the issue with the cache and causing the system to lock up (dead lock) looks to be related specifically. The drive is ... effectively getting too much IO and is having issues handling it. And it's causing issues with reads, including pinning NTFS metadata (which would include volume/disk size info).

Once that issue is resolved, we will make sure that the resizing functionality is working properly, and fix the issue if it's still happening.

thnz · July 28, 2015

Have you guys found the problem yet? Would more dumps help, or is it now a matter of implementing a fix?

Christopher (Drashna) · July 29, 2015

Alex is actively working on the issue, but it is a very, very complicated issue.

I'll bug him about it, though.

Also, the above link (this one: https://stablebit.com/Admin/IssueAnalysis/17719)is where alex will post any status updates, including whien it's fixed (as well as the version number of the fixed version)

And I'll definitely let you know as soon as we have a fix for it.

thnz · July 29, 2015

A shame it turned out to be so complex. Great to know that its being worked on though. Good luck!

Christopher (Drashna) · July 29, 2015

A shame it turned out to be so complex. Great to know that its being worked on though. Good luck!

Definitely. And it's always frustrating on our end when it turns out to be something like that.

thnz · August 2, 2015

Do you know if data loss/corruption can occur when this happens?

I copied several hundred 500mb-1gb files onto a cloud disk, and during upload this deadlock thing happened twice. Once uploading had finished, several of the files became corrupted (they had different file hashes than the originals). The corrupted files seemed to be in two distinct clusters when sorted by file modified time (ie. they were copied sequentially). I'm wondering if chunks of these corrupted files were being uploaded at the time of the deadlock, and weren't caught by recovery after a restart. Is it possible that chunks being uploaded when this crash occurs can become corrupted?

Christopher (Drashna) · August 3, 2015

It shouldn't corrupt it.

However if the system crashed/BSODed, then it's definitely possible (as there is always a possibility of corruption in this case).

However, a chkdsk pass of the CloudDrive should detect and correct any issues

Christopher (Drashna) · August 11, 2015

Anyone seeing this issue, please try the latest beta build:

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.325_x64_BETA.exe

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.0.325_x86_BETA.exe

Alex (the developer) has significantly overhauled the caching system, and it should work much better now. Additionally, when the cache drive is running out of space, it will throttle reads and writes, (and block rights if it gets too slow).

There are some other fixes (including to Amazon Cloud Drive handling of throttling codes, but it may have issues due to authorization issues).

However, we do recommend uninstalling the current version of CloudDrive, deleting the contents of "C:\Program Files\StableBit\CloudDrive" and then installing the new version (due to an installer issue we've identified recently).

thnz · August 11, 2015

Great to see progress on this. Unfortunately I'm still unable to reattach drives in this build (bug in other thread), so can't really give it a proper testing yet.

Christopher (Drashna) · August 11, 2015

Great to see progress on this. Unfortunately I'm still unable to reattach drives in this build (bug in other thread), so can't really give it a proper testing yet.

Definitely. Alex has been devoting a lot of time to trying to fix this issue, because it's a huge one, unfortunately.

As for the reattach bug, I believe I've flagged that for Alex already, and we'll look into it soon.

dragon2611 · August 11, 2015

I'll install this build on the machine I was trying to use last time and see if it solves the issue.

Edit: Installer sat for a very long time and eventually stated the service failed to start, X64 version on server 2012r2

from the event log.

Service cannot be started. System.TypeInitializationException: The type initializer for 'CloudDriveService.Main' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Cove.Util, Version=1.0.5689.32472, Culture=neutral, PublicKeyToken=4c28787c0620f424' or one of its dependencies. The system cannot find the file specified.
   at CloudDriveService.Main..cctor()
   --- End of inner exception stack trace ---
   at CloudDriveService.Main.#vt()
   at #eae.#Efc.#Bfc(String[] #qRc)
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

thnz · August 12, 2015

Just had the 'deadlock' lockup again. Happened while drivepool was duplicating files to the clouddrive. Will upload a dump shortly.

*edit* Uploaded here. Drive is doing recovery right now.

Sign In

I/O deadlock?

Question

Top Posters For This Question

Popular Days

Top Posters For This Question

Popular Days

Popular Posts

Christopher (Drashna)

Christopher (Drashna)

Christopher (Drashna)

Posted Images

130 answers to this question

Recommended Posts

Join the conversation

Announcements