Jump to content
Covecube Inc.
  • Announcements

    • Christopher (Drashna)

      Login issues   11/07/17

      If you have issues with logging in, make sure you use your display name and not the "username" or email.  Or head here for more info.   http://community.covecube.com/index.php?/topic/3252-login-issues/  
    • Christopher (Drashna)

      Getting Help   11/07/17

      If you're experiencing problems with the software, the best way to get ahold of us is to head to https://stablebit.com/Contact, especially if this is a licensing issue.    Issues submitted there are checked first, and handled more aggressively. So, especially if the problem is urgent, please head over there first. 

Search the Community

Showing results for tags 'bsod'.



More search options

  • Search By Tags

    Type tags separated by commas.
  • Search By Author

Content Type


Forums

  • Covecube Inc.
    • Announcements
    • Nuts & Bolts
  • BitFlock
    • General
  • StableBit Scanner
    • General
    • Compatibility
    • Nuts & Bolts
  • StableBit DrivePool
    • General
    • Hardware
    • Nuts & Bolts
  • StableBit CloudDrive
    • General
    • Providers
    • Nuts & Bolts
  • Other
    • Off-topic

Found 6 results

  1. Windows 10: BSOD caused by covefs.sys

    Hi, I've been having a few BSODs recently caused by DrivePool. I'm running the latest Windows 10 creators' update, v10.0.16299 - I believe this only started happening after I updated to it. DrivePool is v2.2.0.852. In this instance I was working on my work PC over remote desktop, saved a file into my DropBox folder which synced it back to my PC (located on my pool) , and it instantly crashed ntfs.sys. I also had this the other week when I was trying to save an excel file locally on my desktop (also located on my pool). If I move my DropBox folder outside my pool it works fine. Here's the crash dump, let me know if you want me to send you the full .dmp.: Crash Dump Analysis provided by OSR Open Systems Resources, Inc. (http://www.osr.com) Online Crash Dump Analysis Service See http://www.osronline.com for more information Windows 8 Kernel Version 16299 MP (4 procs) Free x64 Product: WinNt, suite: TerminalServer SingleUserTS Built by: 16299.15.amd64fre.rs3_release.170928-1534 Machine Name: Kernel base = 0xfffff800`23a83000 PsLoadedModuleList = 0xfffff800`23de4fd0 Debug session time: Sun Nov 12 17:30:08.591 2017 (UTC - 5:00) System Uptime: 6 days 22:32:59.653 ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* SYSTEM_SERVICE_EXCEPTION (3b) An exception happened while executing a system service routine. Arguments: Arg1: 00000000c0000005, Exception code that caused the bugcheck Arg2: fffff8053a9b9215, Address of the instruction which caused the bugcheck Arg3: ffffb7094451d4a0, Address of the context record for the exception that caused the bugcheck Arg4: 0000000000000000, zero. Debugging Details: ------------------ TRIAGER: Could not open triage file : e:\dump_analysis\program\triage\modclass.ini, error 2 EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s". FAULTING_IP: NTFS!NtfsDecodeFileObject+49 fffff805`3a9b9215 8b4804 mov ecx,dword ptr [rax+4] CONTEXT: ffffb7094451d4a0 -- (.cxr 0xffffb7094451d4a0) rax=0000000000000000 rbx=ffff82090fd7a4a0 rcx=0000000000000000 rdx=ffffb7094451e118 rsi=ffffcc0f31c1b590 rdi=ffff82090ac464b8 rip=fffff8053a9b9215 rsp=ffffb7094451de90 rbp=0000000000000000 r8=ffffb7094451df48 r9=ffffb7094451df10 r10=ffffb7094451e0b8 r11=ffff82090ac464b8 r12=0000000000000005 r13=ffffcc0f3bb6de50 r14=ffffb7094451e118 r15=ffff82090fd7a4a0 iopl=0 nv up ei pl nz na pe nc cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202 NTFS!NtfsDecodeFileObject+0x49: fffff805`3a9b9215 8b4804 mov ecx,dword ptr [rax+4] ds:002b:00000000`00000004=00000000 Resetting default scope CUSTOMER_CRASH_COUNT: 1 DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT BUGCHECK_STR: 0x3B PROCESS_NAME: Dropbox.exe CURRENT_IRQL: 0 LAST_CONTROL_TRANSFER: from fffff8053aab994c to fffff8053a9b9215 STACK_TEXT: ffffb709`4451de90 fffff805`3aab994c : ffff8209`0a2018d0 ffff8209`0a201750 ffff8209`0a201830 fffff805`399969da : NTFS!NtfsDecodeFileObject+0x49 ffffb709`4451ded0 fffff805`3aab7091 : 00000000`00000000 00000000`00000000 ffff8208`febe44f0 ffff8208`febee550 : NTFS!NtfsFindTargetElements+0x48 ffffb709`4451df40 fffff805`3aad2cef : ffff8209`0ac46401 ffff8209`0ac464b8 ffffcc0f`00000000 ffff8208`00000000 : NTFS!NtfsSetRenameInfo+0x2d5 ffffb709`4451e340 fffff805`3aad2488 : ffff8209`0ac464b8 ffff8209`17ef2370 ffff8209`17ef2300 ffff8209`0d4d3a00 : NTFS!NtfsCommonSetInformation+0x76f ffffb709`4451e420 fffff800`23ad9339 : ffff8209`0a0d91e0 ffff8209`17ef2370 ffff8209`0ac464b8 ffffb709`4451e460 : NTFS!NtfsFsdSetInformation+0xd8 ffffb709`4451e4c0 fffff805`3c105473 : ffff8209`0a0d91e0 ffff8209`17b620d0 00000000`00000000 ffff8209`17b620d0 : nt!IofCallDriver+0x59 ffffb709`4451e500 ffff8209`0a0d91e0 : ffff8209`17b620d0 00000000`00000000 ffff8209`17b620d0 ffff8209`00000078 : covefs+0x55473 ffffb709`4451e508 ffff8209`17b620d0 : 00000000`00000000 ffff8209`17b620d0 ffff8209`00000078 fffff805`3c0e3c10 : 0xffff8209`0a0d91e0 ffffb709`4451e510 00000000`00000000 : ffff8209`17b620d0 ffff8209`00000078 fffff805`3c0e3c10 00000000`00000000 : 0xffff8209`17b620d0 FOLLOWUP_IP: covefs+55473 fffff805`3c105473 898424e8000000 mov dword ptr [rsp+0E8h],eax SYMBOL_STACK_INDEX: 6 SYMBOL_NAME: covefs+55473 FOLLOWUP_NAME: MachineOwner MODULE_NAME: covefs IMAGE_NAME: covefs.sys DEBUG_FLR_IMAGE_TIMESTAMP: 59bb1a02 STACK_COMMAND: .cxr 0xffffb7094451d4a0 ; kb FAILURE_BUCKET_ID: X64_0x3B_covefs+55473 BUCKET_ID: X64_0x3B_covefs+55473 Followup: MachineOwner --------- Thanks
  2. I spend almost 3hours every day trying to keep my CloudDrive+CloudPool setup alive. For me its a hard job with crashe and unresponsive system and GUIs. ReFS, NTFS, going virutal, going BETA, Clustersize, adding SSD, SSD Addon, cachesize... trying different providers, Dropbox, Azure, FTP, Google Drive, Local... Nothing keeps CloudDrives working more then some hours. But my latest problem is that im sleeping to long so CloudDrive removes my haddrives after dual bluescreens. When my drives are getting removed i see: [CloudDrives] Drive <ID> was previously unmountable. User is not retrying mount. Skipping. 10 drives is now 5 drives.. And now i need to force it....? So... what to do know? Start all over again? ok, this was not clouddrive related, i lost 2SSD´s with the last BSOD... sorry! but can i copy the cache folder to a new drive if i csn recover the data?
  3. BSOD Creating Symlink/Junction on Pool

    Is anybody else getting a blue screen of death creating a symlink or junction on a pool in the latest betas? I'm running Server 2012 R2 and build 615 of DrivePool and I get a SYSTEM_SERVICE_EXCEPTION whenever I try to create a junction or symlink. I reverted back to the last official release (build 561) and it works correctly again. Not sure on what build the BSOD behavior started. Getting ready to keep trying newer versions until the the problem recurs. Just wondering if it's a problem isolated to me and my setup or if this is a more general problem.
  4. Hi all, I just started evaluating Stablebit DP after having read in many places that it was superior to Storage Spaces. I'm equally new on this forum and nor very familiar with how to properly make these posts, what kind of info that may be relevant to to add initially or so. So I'll give it a try and hopefully I can supply any other relevant info on request. Situation and problem: I created a DP on my recently built Win8.1 machine. The pool consists of two 4TB WD Red. Brand new. OS runs on an Intel 120GB SSD. Brand new. I have started loading data on to the pool from external drives. This has worked fine. I can open documents and view images. That works fine. But when I try to open movies it halts for a sec right after the software launches and then I get the BSOD. Same thing happens when I try to create backups of drives/computers. After only a couple of seconds of writing it's the BSOD. Thus, is there seemingly a correlation between intense read/write activities and what causes the BSOD?. The message in the BSOD is along the lines of "kernel mode exception not handled". I've obviously goggled it a little but really only found that it seems to be a very generic type of message. I have also installed Stable bit Scanner after this occurred the first time. No indications of any HW related issues. So now I'm turning to you for help. I couldn't find any other topic on this, please forgive me if I missed something already covering this. In either case I'd be very happy to be pointed in the right direction for how to solve it. Thanks a bunch in advance!
  5. Ouch! BSOD while installing 2.0.0.265! No share access, nothing running in the background, just doing the update by remote desktop and having one explorer opened on the pool. I uploaded the memory dump. Edit: now the DP install is bad. Impossible to uninstall, impossible to update. Here is the log from the installer (cannot we upload files to the forum?) [11B4:11B8][2013-06-08T14:23:43]i001: Burn v3.7.1224.0, Windows v6.2 (Build 9200: Service Pack 0), path: \\GOLGOTH\Outils\Server Resources\Tools\StableBit\StableBit.DrivePool_2.0.0.265_x64_BETA.exe, cmdline: '-burn.unelevated BurnPipe.{7E4311D7-B505-4EBD-BCC1-4DC6FF8E0E1A} {6DE7F554-71F4-4E8E-9D3F-7C36BACB8FD8} 4520' [11B4:11B8][2013-06-08T14:23:43]i000: Initializing string variable 'InstallFolder' to value '[PlatformProgramFilesFolder]\StableBit\DrivePool' [11B4:11B8][2013-06-08T14:23:43]i000: Setting string variable 'WixBundleLog' to value 'C:\Users\ADMINI~1\AppData\Local\Temp\StableBit_DrivePool_(64_bit)_20130608142343.log' [11B4:11B8][2013-06-08T14:23:43]i000: Setting string variable 'WixBundleOriginalSource' to value '\\GOLGOTH\Outils\Server Resources\Tools\StableBit\StableBit.DrivePool_2.0.0.265_x64_BETA.exe' [11B4:11B8][2013-06-08T14:23:44]i000: Setting string variable 'WixBundleName' to value 'StableBit DrivePool (64 bit)' [11B4:11B8][2013-06-08T14:23:44]i100: Detect begin, 2 packages [11B4:11B8][2013-06-08T14:23:44]i000: Setting string variable 'NETFRAMEWORK40CLIENT' to value '1' [11B4:11B8][2013-06-08T14:23:44]i052: Condition 'NOT VersionNT64' evaluates to false. [11B4:11B8][2013-06-08T14:23:44]i052: Condition 'VersionNT64' evaluates to true. [11B4:11B8][2013-06-08T14:23:44]i000: Setting string variable 'PlatformProgramFilesFolder' to value 'C:\Program Files' [11B4:11B8][2013-06-08T14:23:44]i000: Registry key not found. Key = 'SOFTWARE\StableBit\DrivePool' [11B4:11B8][2013-06-08T14:23:44]i000: Setting version variable 'KmdfVersion' to value '1.11.9200.16384' [11B4:11B8][2013-06-08T14:23:44]i000: Product not found: {1AB7D699-E19A-45E6-992C-2FB2DC1730C7} [11B4:11B8][2013-06-08T14:23:44]i000: Setting version variable 'DrivePool1Version' to value '0.0.0.0' [11B4:11B8][2013-06-08T14:23:44]i102: Detected related bundle: {e0faf431-d1db-4b00-ab2f-f6bd880f001b}, type: Upgrade, scope: PerMachine, version: 2.0.260.0, operation: MajorUpgrade [11B4:11B8][2013-06-08T14:23:44]i052: Condition 'NETFRAMEWORK40CLIENT' evaluates to true. [11B4:11B8][2013-06-08T14:23:44]i103: Detected related package: {43053F4F-87D0-4459-A600-03896AC6F486}, scope: PerMachine, version: 2.0.260.0, language: 1033 operation: MinorUpdate [11B4:11B8][2013-06-08T14:23:44]i101: Detected package: NetFx40ClientWeb, state: Present, cached: None [11B4:11B8][2013-06-08T14:23:44]i101: Detected package: DrivePoolApplication, state: Present, cached: Complete [11B4:11B8][2013-06-08T14:23:44]i052: Condition '(VersionNT >= v6.0)' evaluates to true. [11B4:11B8][2013-06-08T14:23:44]i052: Condition '(KmdfVersion >= v1.9)' evaluates to true. [11B4:11B8][2013-06-08T14:23:44]i052: Condition 'DrivePool1Version = v0.0' evaluates to true. [11B4:11B8][2013-06-08T14:23:44]i199: Detect complete, result: 0x0 [11B4:11B8][2013-06-08T14:23:46]i200: Plan begin, 2 packages, action: Install [11B4:11B8][2013-06-08T14:23:46]w321: Skipping dependency registration on package with no dependency providers: NetFx40ClientWeb [11B4:11B8][2013-06-08T14:23:46]i000: Setting string variable 'WixBundleLog_DrivePoolApplication' to value 'C:\Users\ADMINI~1\AppData\Local\Temp\StableBit_DrivePool_(64_bit)_20130608142343_0_DrivePoolApplication.log' [11B4:11B8][2013-06-08T14:23:46]i201: Planned package: NetFx40ClientWeb, state: Present, default requested: Present, ba requested: Present, execute: None, rollback: None, cache: No, uncache: No, dependency: None [11B4:11B8][2013-06-08T14:23:46]i201: Planned package: DrivePoolApplication, state: Present, default requested: Present, ba requested: Present, execute: MinorUpgrade, rollback: None, cache: No, uncache: No, dependency: Register [11B4:11B8][2013-06-08T14:23:46]i207: Planned related bundle: {e0faf431-d1db-4b00-ab2f-f6bd880f001b}, type: Upgrade, default requested: Absent, ba requested: Absent, execute: Uninstall, rollback: Install, dependency: None [11B4:11B8][2013-06-08T14:23:46]i299: Plan complete, result: 0x0 [11B4:11B8][2013-06-08T14:23:46]i300: Apply begin [11A8:11AC][2013-06-08T14:23:46]i360: Creating a system restore point. [11A8:11AC][2013-06-08T14:23:46]i362: System restore disabled, system restore point not created. [11A8:11AC][2013-06-08T14:23:46]i000: Caching bundle from: 'C:\Users\ADMINI~1\AppData\Local\Temp\{f5ded207-a61f-453d-befa-a062d2c0bc7d}\.be\StableBit.DrivePool_x64.exe' to: 'C:\ProgramData\Package Cache\{f5ded207-a61f-453d-befa-a062d2c0bc7d}\StableBit.DrivePool_x64.exe' [11A8:11AC][2013-06-08T14:23:46]i320: Registering bundle dependency provider: {f5ded207-a61f-453d-befa-a062d2c0bc7d}, version: 2.0.265.0 [11A8:11C8][2013-06-08T14:23:46]i304: Verified existing payload: DrivePoolApplication at path: C:\ProgramData\Package Cache\{43053F4F-87D0-4459-A600-03896AC6F486}v2.0.265\StableBit.DrivePool.msi. [11A8:11AC][2013-06-08T14:23:46]i323: Registering package dependency provider: {43053F4F-87D0-4459-A600-03896AC6F486}, version: 2.0.265, package: DrivePoolApplication [11A8:11AC][2013-06-08T14:23:46]i301: Applying execute package: DrivePoolApplication, action: MinorUpgrade, path: C:\ProgramData\Package Cache\{43053F4F-87D0-4459-A600-03896AC6F486}v2.0.265\StableBit.DrivePool.msi, arguments: ' ARPSYSTEMCOMPONENT="1" MSIFASTINSTALL="7"' [11A8:11AC][2013-06-08T14:23:49]e000: Error 0x80070643: Failed to perform minor upgrade of MSI package. [11A8:11AC][2013-06-08T14:23:49]e000: Error 0x80070643: Failed to execute MSI package. [11B4:11B8][2013-06-08T14:23:49]e000: Error 0x80070643: Failed to configure per-machine MSI package. [11B4:11B8][2013-06-08T14:23:49]i319: Applied execute package: DrivePoolApplication, result: 0x80070643, restart: None [11B4:11B8][2013-06-08T14:23:49]e000: Error 0x80070643: Failed to execute MSI package. [11A8:11AC][2013-06-08T14:23:49]i329: Removed package dependency provider: {43053F4F-87D0-4459-A600-03896AC6F486}, package: DrivePoolApplication [11A8:11AC][2013-06-08T14:23:49]i330: Removed bundle dependency provider: {f5ded207-a61f-453d-befa-a062d2c0bc7d} [11A8:11AC][2013-06-08T14:23:49]i352: Removing cached bundle: {f5ded207-a61f-453d-befa-a062d2c0bc7d}, from path: C:\ProgramData\Package Cache\{f5ded207-a61f-453d-befa-a062d2c0bc7d}\ [11B4:11B8][2013-06-08T14:23:50]i399: Apply complete, result: 0x80070643, restart: None, ba requested restart: No Edit 2: currently restoring my server from backup. Nothing worked, cannot uninstall DP, tried in safe mode and all the same the installer drop me. Tried renaming DP install folder, but when restarting the pool drive is still here. So many problems with this software... not sure to continue with it.
  6. Continuing the thread from the old forum: http://forum.covecube.com/discussion/1129/critical-bsod-when-accessing-dp-over-a-network-share Just to recap, I've received a number of memory dumps over the past month or so showing a system crash in srv2.sys. Srv2.sys is Microsoft's file sharing driver that translates network file I/O requests into local file I/O requests on the server, but the implication is of course that StableBit DrivePool may somehow be the cause of these crashes. The crashes only occur on Windows 8 / Windows Server 2012 (including Essentials). Paris has submitted the best dump to date on this issue and I've analyzed it in detail. I'm going to post a technical description of the crash for anyone who can read this sort of thing. What we have A full memory dump of the crash. Verifier enabled on all drivers at the time of the crash (giving us additional data about the crash). ETW logging enabled on CoveFS / CoveFSDisk, giving us everything that CoveFS did right before the crash. The system 3: kd> vertarget Windows 8 Kernel Version 9200 MP (4 procs) Free x64 Product: Server, suite: TerminalServer DataCenter SingleUserTS Built by: 9200.16581.amd64fre.win8_gdr.130410-1505 Kernel base = 0xfffff800`65214000 PsLoadedModuleList = 0xfffff800`654e0a20 Debug session time: Fri May 31 04:45:08.610 2013 (UTC - 4:00) System Uptime: 0 days 0:21:01.550 3: kd> !sysinfo cpuinfo [CPU Information] ~MHz = REG_DWORD 3100 Component Information = REG_BINARY 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 Configuration Data = REG_FULL_RESOURCE_DESCRIPTOR ff,ff,ff,ff,ff,ff,ff,ff,0,0,0,0,0,0,0,0 Identifier = REG_SZ Intel64 Family 6 Model 58 Stepping 9 ProcessorNameString = REG_SZ Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz Update Status = REG_DWORD 2 VendorIdentifier = REG_SZ GenuineIntel MSR8B = REG_QWORD ffffffff00000000 The crash 3: kd> !Analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* PAGE_FAULT_IN_NONPAGED_AREA (50) Invalid system memory was referenced. This cannot be protected by try-except, it must be protected by a Probe. Typically the address is just plain bad or it is pointing at freed memory. Arguments: Arg1: fffffa8303aa41ea, memory referenced. Arg2: 0000000000000000, value 0 = read operation, 1 = write operation. Arg3: fffff8800556e328, If non-zero, the instruction address which referenced the bad memory address. Arg4: 0000000000000000, (reserved) Debugging Details: ------------------ READ_ADDRESS: fffffa8303aa41ea Nonpaged pool FAULTING_IP: srv2!Smb2ContinueUncachedRead+26c28 fffff880`0556e328 410fb644240a movzx eax,byte ptr [r12+0Ah] MM_INTERNAL_CODE: 0 IMAGE_NAME: srv2.sys DEBUG_FLR_IMAGE_TIMESTAMP: 51637dde MODULE_NAME: srv2 FAULTING_MODULE: fffff880054ff000 srv2 DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT BUGCHECK_STR: AV PROCESS_NAME: System CURRENT_IRQL: 0 TRAP_FRAME: fffff88004d3fb60 -- (.trap 0xfffff88004d3fb60) NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=fffffa830396c150 rbx=0000000000000000 rcx=0000000000000000 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000 rip=fffff8800556e328 rsp=fffff88004d3fcf0 rbp=fffff9801e9e0af0 r8=00000000000006e5 r9=fffff88005544680 r10=fffffa83035a6c40 r11=0000000000000001 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei ng nz na po cy srv2!Smb2ContinueUncachedRead+0x26c28: fffff880`0556e328 410fb644240a movzx eax,byte ptr [r12+0Ah] ds:00000000`0000000a=?? Resetting default scope LAST_CONTROL_TRANSFER: from fffff8006532f3f1 to fffff8006526e440 STACK_TEXT: fffff880`04d3f978 fffff800`6532f3f1 : 00000000`00000050 fffffa83`03aa41ea 00000000`00000000 fffff880`04d3fb60 : nt!KeBugCheckEx fffff880`04d3f980 fffff800`652a8acb : 00000000`00000000 fffffa83`03aa41ea fffffa83`035be040 fffff880`03cc4419 : nt! ?? ::FNODOBFM::`string'+0x33c2b fffff880`04d3fa20 fffff800`6526beee : 00000000`00000000 fffff980`254b6950 fffff980`1e9e0b00 fffff880`04d3fb60 : nt!MmAccessFault+0x55b fffff880`04d3fb60 fffff880`0556e328 : 00000000`00000000 fffff880`00000000 ffff2f2d`390b1a54 fffff980`01dc8f20 : nt!KiPageFault+0x16e fffff880`04d3fcf0 fffff880`055470de : fffffa83`03c3d1e0 fffff980`01dc8f20 fffff980`254b6950 fffffa83`01f99040 : srv2!Smb2ContinueUncachedRead+0x26c28 fffff880`04d3fd50 fffff880`055455bd : 00000000`00000002 fffffa83`01f99040 fffff980`254b6c60 fffff800`6524acbe : srv2!Smb2ExecuteRead+0x6ce fffff880`04d3fde0 fffff880`05545a64 : fffffa83`0084cd18 fffff980`254b6950 fffff980`1b44efd0 fffff980`254b6950 : srv2!Smb2ExecuteProviderCallback+0x6d fffff880`04d3fe50 fffff880`05543180 : fffff980`1b54cd80 fffff980`1b54cd80 00000000`00000001 fffff980`254b6950 : srv2!SrvProcessPacket+0xed fffff880`04d3ff10 fffff800`65268b27 : fffff880`04d3ff01 00000000`00000000 fffff980`254b6960 fffff880`05546000 : srv2!SrvProcpWorkerThreadProcessWorkItems+0x171 fffff880`04d3ff80 fffff800`65268aed : fffff980`1b54cd01 00000000`0000c000 00000000`00000003 fffff800`652c3ab8 : nt!KxSwitchKernelStackCallout+0x27 fffff880`07f9c9e0 fffff800`652c3ab8 : fffffa83`00000012 fffff980`1b54cd01 00000000`00000006 fffff880`07f97000 : nt!KiSwitchKernelStackContinue fffff880`07f9ca00 fffff800`652c63f5 : fffff880`05543010 fffff980`1b54cd80 fffff980`1b54cd00 fffff980`00000000 : nt!KeExpandKernelStackAndCalloutInternal+0x218 fffff880`07f9cb00 fffff880`05500da5 : fffff980`1b54cd80 fffff980`1b54cd00 fffff980`1b54cd80 fffff800`65277cc4 : nt!KeExpandKernelStackAndCalloutEx+0x25 fffff880`07f9cb40 fffff800`652ac2b1 : fffffa83`035be040 fffff880`05546000 fffff980`1b54cde0 fffff900`00000000 : srv2!SrvProcWorkerThreadCommon+0x75 fffff880`07f9cb80 fffff800`65241045 : fffffa83`03058660 00000000`00000080 fffff800`652ac170 fffffa83`035be040 : nt!ExpWorkerThread+0x142 fffff880`07f9cc10 fffff800`652f5766 : fffff800`6550c180 fffffa83`035be040 fffff800`65566880 fffffa83`0088d980 : nt!PspSystemThreadStartup+0x59 fffff880`07f9cc60 00000000`00000000 : fffff880`07f9d000 fffff880`07f97000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16 STACK_COMMAND: kb FOLLOWUP_IP: srv2!Smb2ContinueUncachedRead+26c28 fffff880`0556e328 410fb644240a movzx eax,byte ptr [r12+0Ah] SYMBOL_STACK_INDEX: 4 SYMBOL_NAME: srv2!Smb2ContinueUncachedRead+26c28 FOLLOWUP_NAME: MachineOwner BUCKET_ID_FUNC_OFFSET: 26c28 FAILURE_BUCKET_ID: AV_VRF_srv2!Smb2ContinueUncachedRead BUCKET_ID: AV_VRF_srv2!Smb2ContinueUncachedRead Followup: MachineOwner --------- From the auto analysis we can see that the memory address 0xfffffa8303aa41ea was being accessed from some code at address 0xfffff8800556e328. We can also see that the function that crashed the system is srv2!Smb2ContinueUncachedRead. We can check that memory address and it is indeed invalid: 3: kd> dd 0xfffffa8303aa41ea fffffa83`03aa41ea ???????? ???????? ???????? ???????? fffffa83`03aa41fa ???????? ???????? ???????? ???????? fffffa83`03aa420a ???????? ???????? ???????? ???????? fffffa83`03aa421a ???????? ???????? ???????? ???????? fffffa83`03aa422a ???????? ???????? ???????? ???????? fffffa83`03aa423a ???????? ???????? ???????? ???????? fffffa83`03aa424a ???????? ???????? ???????? ???????? fffffa83`03aa425a ???????? ???????? ???????? ???????? What was srv2 trying to do? So the next question to ask is what was srv2 trying to do and why did it fail? I've gone ahead and decompiled the portion of srv2 that is causing the crash, and here it is: if ( mdl1 && mdl1 != mdl2 && !(mdl1->MdlFlags & MDL_SOURCE_IS_NONPAGED_POOL) ) { do { mdlCurrent = mdl1; mdlFlags = mdl1->MdlFlags; mdl1 = mdl1->Next; if ( mdlFlags & (MDL_PARTIAL_HAS_BEEN_MAPPED | MDL_PAGES_LOCKED) ) { MmUnlockPages(mdlCurrent); } IoFreeMdl(mdlCurrent); } while ( mdl1 ); *(_DWORD *)(Length + 4) = 0; } A MDL is a kernel structure that simply describes some memory (http://msdn.microsoft.com/en-us/library/windows/hardware/ff554414(v=vs.85).aspx). The MDL variables: mdl1: 0xfffffa83`03aa41e0 (invalid memory pointer) mdl2: 0xfffffa83`03c3d1e0 3: kd> dt nt!_MDL fffffa83`03c3d1e0 +0x000 Next : (null) +0x008 Size : 0n568 +0x00a MdlFlags : 0n4 +0x00c AllocationProcessorNumber : 0xf980 +0x00e Reserved : 0xffff +0x010 Process : (null) +0x018 MappedSystemVa : 0xfffffa83`03bfd000 Void +0x020 StartVa : 0xfffffa83`03bfd000 Void +0x028 ByteCount : 0x40150 +0x02c ByteOffset : 0 The crash occurs at the point when the function tries to access the MdlFlags member of mdl1 (mdl1->MdlFlags). Since mdl1 points to an invalid memory address, we can't read the flags in. The assembly instructions look like this: srv2!Smb2ContinueUncachedRead+0x26c28: fffff880`0556e328 410fb644240a movzx eax,byte ptr [r12+0Ah] fffff880`0556e32e a804 test al,4 fffff880`0556e330 0f853294fdff jne srv2!Smb2ContinueUncachedRead+0x68 (fffff880`05547768) r12 is mdl1, and we crash when trying to read in the flags. The connection to Fast I/O In every single crash dump that I've seen, the crash always occurs after a successful (non-waiting) Fast I/O read. In fact, the function that calls the crashing function (srv2!Smb2ExecuteRead+0x6ce) has an explicit condition to test for this. Where did mdl1 go? So the question is, why is mdl1 invalid? Did it exist before and was freed, or was there some kind of memory corruption? Here are my observations on this: In every dump that I've seen, the addresses look right. What I mean by that is that the seemingly invalid mdl1 address falls roughly into the same address range as mdl2. It always starts correctly and always ends with 1e0. If this crash was due to faulty RAM, then I would expect to see this address fluctuate wildly. The crash always occurs in the same place (plus or minus a few lines of code). To me, this indicates that there is a bug somewhere. Based on these observations I'm assuming that the mdl1 address in indeed valid, and so it must have been previously freed. But who freed it? We can answer that with a simple verifier query: 3: kd> !verifier 0x80 fffffa8303aa41e0 Log of recent kernel pool Allocate and Free operations: There are up to 0x10000 entries in the log. Parsing 0x0000000000010000 log entries, searching for address 0xfffffa8303aa41e0. ====================================================================== Pool block fffffa83`03aa3000, Size 00000000000018e0, Thread fffff80065566880 fffff80065864a32 nt!VfFreePoolNotification+0x4a fffff80065486992 nt!ExFreePool+0x8a0 fffff80065855597 nt!VerifierExFreePoolWithTag+0x47 fffff880013b32bf vmbkmcl!VmbChannelPacketComplete+0x1df fffff88003f91997 netvsc63!NvscMicroportCompleteMessage+0x67 fffff88003f916a3 netvsc63!ReceivePacketMessage+0x1e3 fffff88003f913ff netvsc63!NvscKmclProcessPacket+0x23f fffff880013b2844 vmbkmcl!InpProcessQueue+0x164 fffff880013b402f vmbkmcl!InpFillAndProcessQueue+0x6f fffff880013b7cb6 vmbkmcl! ?? ::FNODOBFM::`string'+0xb16 fffff880014790d7 vmbus!ChildInterruptDpc+0xc7 fffff80065296ca1 nt!KiExecuteAllDpcs+0x191 fffff800652968e0 nt!KiRetireDpcList+0xd0 ====================================================================== Pool block fffffa8303aa3000, Size 00000000000018d0, Thread fffff80065566880 fffff80065855a5d nt!VeAllocatePoolWithTagPriority+0x2d1 fffff88001058665 VerifierExt!ExAllocatePoolWithTagPriority_internal_wrapper+0x49 fffff80065855f02 nt!VerifierExAllocatePoolEx+0x2a fffff880013b2681 vmbkmcl!InpFillQueue+0x641 fffff880013b4004 vmbkmcl!InpFillAndProcessQueue+0x44 fffff880013b7cb6 vmbkmcl! ?? ::FNODOBFM::`string'+0xb16 fffff880014790d7 vmbus!ChildInterruptDpc+0xc7 fffff80065296ca1 nt!KiExecuteAllDpcs+0x191 fffff800652968e0 nt!KiRetireDpcList+0xd0 fffff800652979ba nt!KiIdleLoop+0x5a ====================================================================== The memory has been originally allocated by vmbkmcl.sys, and has already been freed at the point of the crash. Googling, I found that vmbkmcl.sys is a "Hyper-V VMBus KMCL", and netvsc63.sys is the "Virtual NDIS6.3 Miniport". File times Here are the file times of the drivers that are involved in this complicated interaction. 3: kd> lmvm srv2 start end module name fffff880`054ff000 fffff880`055a0000 srv2 (private pdb symbols) c:\symbols\srv2.pdb\B796522F4D804083998D25552950C4202\srv2.pdb Loaded symbol image file: srv2.sys Image path: \SystemRoot\System32\DRIVERS\srv2.sys Image name: srv2.sys Timestamp: Mon Apr 08 22:33:02 2013 (51637DDE) CheckSum: 000A6B64 ImageSize: 000A1000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4 3: kd> lmvm vmbkmcl start end module name fffff880`013b0000 fffff880`013c6000 vmbkmcl (pdb symbols) c:\symbols\vmbkmcl.pdb\82188957E5784EDD91906B760767302E1\vmbkmcl.pdb Loaded symbol image file: vmbkmcl.sys Image path: \SystemRoot\System32\drivers\vmbkmcl.sys Image name: vmbkmcl.sys Timestamp: Wed Jul 25 22:28:33 2012 (5010AB51) CheckSum: 000250C9 ImageSize: 00016000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4 3: kd> lmvm netvsc63 start end module name fffff880`03f90000 fffff880`03faa000 netvsc63 (private pdb symbols) c:\symbols\netvsc63.pdb\BD38B199A4C94771860A5F2390CC30E61\netvsc63.pdb Loaded symbol image file: netvsc63.sys Image path: netvsc63.sys Image name: netvsc63.sys Timestamp: Sat Feb 02 02:23:05 2013 (510CBED9) CheckSum: 0001B2D9 ImageSize: 0001A000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4 Possible sequence of events In short, it seems to me that: Some memory was allocated to process a network request. That memory was passed to srv2.sys, which is processing that request. The original driver has decided that the memory is no longer needed and freed the memory. srv2.sys is ignorantly trying to access the now freed memory. Workarounds As a potential workaround, turning off Fast I/O should prevent the code that is causing the problem from running. DrivePool 2.0 doesn't yet contain a setting for this but I'll add it in the next build. Turning on Network I/O Boost should also prevent the problem because we do some extra processing on networked read requests when that is enabled, which bypasses Fast I/O. Connection to DrivePool I'm still trying to find a connection to DrivePool in all of this, but I can't. I still can't reproduce this crash on any of the test servers here (4 of them running the windows 8 kernel), nor can I reproduce this on any of my VMs (using VirtualBox). Fast I/O doesn't deal with MDLs at all, so DrivePool never sees any of the variables that I've talked about here. The Fast I/O code in CoveFS is fairly short and easy to check. Because of the potential Hyper-V connection shown above, I'll try to test for this on a Hyper-V VM. As far as what DrivePool was doing right before the crash, I can see from the in-memory logs that it has just completed a Fast I/O read request and has successfully read in 262,144 bytes. Because I don't have a definitive reproducible case, I can't be 100% certain as to what the issue is. I'll keep digging and will let you guys know if I find anything else.
×