Jump to content
  • 0

Covefs Causing BSOD With Windows Server 2012 Essentials


dbone1026

Question

Hello - I have been using Stablebit DrivePool with my WIndows Server 2012 Essentials Server and frequently get BSODs (which it appears now is related to Covefs).  My server specs are:

 

O/S: Windows Server 2012 Essentials
x64
Full Retail Version
Age of System (hardware): Almost 2 years
Age of OS Installation: Almost 2 years (no reinstallations done)

CPU: Intel Core i5-3570K Ivy Bridge
Video Card: None
RAID Card: 2 x HighPoint RocketRAID 2720SGL 8-Port PCI Express 2.0
Motherboard: Gigabyte G1.Sniper 3 LGA 1155 Intel Z77
Power Supply: Corsair Gold AX850 850W ATX12V v2.31
Memory: G.SKILL Ripjaws X Series 16GB (8x2) 240Pin DDR3 SDRAM DDR3 1600

System Manufacturer: DYI build

Case: Lian Li PC-D8000

 

When the BSOD occurs it is typically the "Special Pool Detected Memory Corrupt" error. I submitted my memory dumps for analysis and the developer believes Covefs is the culprit. Below is the code. As well, this is his determination:

 

 

Well, we know the Kernel covefs driver didn't clean up it's allocations and caused corruption on a page. The pool is corrupt to the point where we cannot check who owned the page prior to the corruption. It's no longer a valid page, thus writing over all information.
Covefs is a Windows Driver Development Kit Kernel driver, I haven't used the kit myself, I have been meaning to give it a try.
I suggest you remove the package that this driver comes with, or the entire kit, whichever option is the most feasible for your machine. You could try an update as it dates back to June 2014, if that doesn't stop the crashes then remove the software.

 

3: kd> .bugcheck
Bugcheck code 000000C1
Arguments fffff980`b4d1ae70 fffff980`b4d1a2da 00000000`00474190 00000000`00000032

//Special pool corruption

3: kd> kn
 # Child-SP          RetAddr           Call Site
00 fffff880`07d85108 fffff802`8c00582a nt!KeBugCheckEx //Bugcheck, corruption was found
01 fffff880`07d85110 fffff802`8c004d54 nt!MiCheckSpecialPoolSlop+0x8a //Check the slop bytes for corruption
02 fffff880`07d85150 fffff802`8c0e97e9 nt!MmFreeSpecialPool+0x178 //Memory manager free special pool function
03 fffff880`07d85280 fffff802`8c4b5577 nt!ExFreePool+0x6d8 //Kernel support routine to free the pool
04 fffff880`07d85360 fffff880`01b6942d nt!VerifierExFreePoolWithTag+0x47 //Free pool verifier hooked function
05 fffff880`07d85390 fffff880`01b77944 covefs+0x342d
06 fffff880`07d853c0 fffff880`01b779ad covefs+0x11944
07 fffff880`07d853f0 fffff880`01b6b951 covefs+0x119ad
08 fffff880`07d85440 fffff880`01b6a600 covefs+0x5951
09 fffff880`07d85520 fffff880`01b6a4ab covefs+0x4600
0a fffff880`07d855a0 fffff880`01b6b21e covefs+0x44ab
0b fffff880`07d85610 fffff880`01b6a6ee covefs+0x521e
0c fffff880`07d85780 fffff880`01b6a4ab covefs+0x46ee
0d fffff880`07d85800 fffff880`01b6b21e covefs+0x44ab
0e fffff880`07d85870 fffff880`01b6a6ee covefs+0x521e
0f fffff880`07d859e0 fffff880`01b6a4ab covefs+0x46ee
10 fffff880`07d85a60 fffff880`01b6b21e covefs+0x44ab
11 fffff880`07d85ad0 fffff880`01b6a6ee covefs+0x521e
12 fffff880`07d85c40 fffff880`01b6a4ab covefs+0x46ee
13 fffff880`07d85cc0 fffff880`01b6b21e covefs+0x44ab
14 fffff880`07d85d30 fffff880`01b6a6ee covefs+0x521e
15 fffff880`07d85ea0 fffff880`01b6a4ab covefs+0x46ee
16 fffff880`07d85f20 fffff880`01b6b21e covefs+0x44ab
17 fffff880`07d85f90 fffff880`01b6a6ee covefs+0x521e
18 fffff880`07d86100 fffff880`01b6a4ab covefs+0x46ee
19 fffff880`07d86180 fffff880`01b6b21e covefs+0x44ab
1a fffff880`07d861f0 fffff880`01b6a6ee covefs+0x521e
1b fffff880`07d86360 fffff880`01b6a4ab covefs+0x46ee
1c fffff880`07d863e0 fffff880`01b6b21e covefs+0x44ab
1d fffff880`07d86450 fffff880`01b6a6ee covefs+0x521e
1e fffff880`07d865c0 fffff880`01b6a4ab covefs+0x46ee
1f fffff880`07d86640 fffff880`01b6b21e covefs+0x44ab
20 fffff880`07d866b0 fffff880`01b6a6ee covefs+0x521e
21 fffff880`07d86820 fffff880`01b6a4ab covefs+0x46ee
22 fffff880`07d868a0 fffff880`01b6b21e covefs+0x44ab
23 fffff880`07d86910 fffff880`01b6ba77 covefs+0x521e
24 fffff880`07d86a80 fffff880`01b6bd67 covefs+0x5a77
25 fffff880`07d86af0 fffff880`01bb1aae covefs+0x5d67
26 fffff880`07d86b90 fffff802`8bea4f69 covefs+0x4baae //Covefs internal functions
27 fffff880`07d86c10 fffff802`8bf592f6 nt!PspSystemThreadStartup+0x59 //Start system thread
28 fffff880`07d86c60 00000000`00000000 nt!KiStartSystemThread+0x16 //Start system thread

3: kd> !address fffff980b4d1ae70
Mapping user range ...
Mapping system range ...
Mapping non addressable range ...
Mapping page tables...
Mapping hyperspace...
Mapping HAL reserved range...
Mapping User Probe Area...
Mapping system shared page...
Mapping system cache working set...
Mapping loader mappings...
Mapping system PTEs...
Mapping system paged pool...
Mapping session space...
Mapping dynamic system space...
Mapping PFN database...
Mapping non paged pool...
Mapping VAD regions...
Mapping module regions...
Mapping process, thread, and stack regions...
Mapping system cache regions...


Usage:                  
Base Address:           fffff980`b4600000
End Address:            fffff980`b5a00000
Region Size:            00000000`01400000
VA Type:                SystemDynamicSpace //Dynamic kernel system space
VAD Address:            0x2c003400360072
Commit Charge:          0x10000013a
Protection:             0x7feea5cff88 []
Memory Usage:           Private
No Change:              yes
More info:              !vad 0xfffff980b4600000

//Corruption to page

fffff980`b4d1ae30  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae40  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae50  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae60  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae70  07d85360 fffff880 5cafce70 fffff980  `S......p..\....
fffff980`b4d1ae80  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1ae90  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1aea0  008a008a 00000000 5ab40f70 fffff980  ........p..Z....

//Full data block for known allocation

fffff980`b4d1ae70  07d85360 fffff880 5cafce70 fffff980  `S......p..\....
fffff980`b4d1ae80  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1ae90  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1aea0  008a008a 00000000 5ab40f70 fffff980  ........p..Z....
fffff980`b4d1aeb0  00000001 00000000 008a002e 00000000  ................
fffff980`b4d1aec0  5ab40f70 fffff980 00000000 00000000  p..Z............
fffff980`b4d1aed0  07d854f0 fffff880 5cafced0 fffff980  .T.........\....
fffff980`b4d1aee0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1aef0  00000000 00000000 008a002e 00000000  ................
fffff980`b4d1af00  5ab40f70 fffff980 008a008a 00000000  p..Z............
fffff980`b4d1af10  5ab40f70 fffff980 01340134 00000000  p..Z....4.4.....
fffff980`b4d1af20  b4bb4ec0 fffff980 00000001 00000000  .N..............
fffff980`b4d1af30  01340134 00000000 b4bb4ec0 fffff980  4.4......N......
fffff980`b4d1af40  00000000 00000000 3a3795ff 00000000  ..........7:....
fffff980`b4d1af50  08f51510 00000000 08f51510 00000000  ................
fffff980`b4d1af60  00000008 00000200 00000007 00000004  ................
fffff980`b4d1af70  00000002 00000000 00000000 00000000  ................
fffff980`b4d1af80  00000000 00000000 00000000 00000000  ................
fffff980`b4d1af90  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afa0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afb0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afc0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afd0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afe0  00000000 00000000 0a606d70 fffff980  ........pm`.....
fffff980`b4d1aff0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1b000  ???????? ???????? ???????? ????????  ????????????????

3: kd> lmvm covefs
start             end                 module name
fffff880`01b66000 fffff880`01bcd000   covefs     (no symbols)           
    Loaded symbol image file: covefs.sys
    Image path: \??\C:\Windows\system32\drivers\covefs.sys
    Image name: covefs.sys
    Timestamp:        Sat Jun 07 19:34:27 2014 (53935B33)
    CheckSum:         0006C805
    ImageSize:        00067000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Any help would be greatly appreciated it DrivePool is the culprit. 

 

Link to comment
Share on other sites

15 answers to this question

Recommended Posts

  • 0

What was the ticket/contact number for this issue?

 

 Yeah, I used the Windows Memory DIagnostic tool and ran into no memory issues.

Basic or extended check? And how many passes?

 

Also, do you have any antivirus installed on the system? And if so, which?

 

 

Also, it may be a good idea to physically inspect the motherboard. Look for bulged capacitors, burn marks, warping, etc.

Link to comment
Share on other sites

  • 0

What was the ticket/contact number for this issue?

 

Basic or extended check? And how many passes?

 

Also, do you have any antivirus installed on the system? And if so, which?

 

 

Also, it may be a good idea to physically inspect the motherboard. Look for bulged capacitors, burn marks, warping, etc.

 

No ticket info. I had submitted my issue on the Sysnative forum for them to look at my memory dump when they noticed CoveFS was running into issues.

 

I just did a basic check. What would you recommend and I can rerun. 

 

No AV installed. 

 

Honestly I have had issues with this build from the start. The first issue was installing the NIC drivers for Intel which Server 2012 would not recognize (appears this was a common issue and was able to find a work around). I am to the point where I would be happy just rebuilding with a new motherboard if we determine it is hardware related.

Link to comment
Share on other sites

  • 0

Upload the dump to us. Anytime you have a BSOD that you even suspect is related to the pool, please do so.

 

The reasons is that only "we" have the driver symbol files for the CoveFS.sys driver. Which means that only we know what is EXACTLY happening at the time of the crash.  While others can guess, it's nowhere near as good. 

And that's what all the "Covefs internal function" references are. It's specific actions that our driver is taking. 

 

So, please do upload the dump to us.

If needed, using the following link:

http://wiki.covecube.com/StableBit_DrivePool_System_Freeze

There is a Box upload widget at the bottom of the page (you may need to disable the ad or script blockers to see it).

 

ALso, if you want, utilities such as "WhoCrashed" or Blue Screen View will also tell you what it thinks caused the crashes, as well. They're great "quick check" utilities.

 

 

As for the check, an extended check is always a better option. It takes much longer, but it's much more thorough.

 

 

 

 

Also, what version of StableBit DrivePool are you using specifically?

Link to comment
Share on other sites

  • 0

Sorry for just responding now, moved this weekend so have been offline.

 

As far as the memory dump, I can't reproduce the memory crash, it just happens when it happens (just happened again a few minutes ago). I do have the memory dump files from the crashes though, can I just upload that or does it have to be the specific method referenced in the wiki you linked to?

 

Also, I am running Drivepool version 2.1.0.558

Link to comment
Share on other sites

  • 0

The dumps are absolutely fine. 

No need to reproduce, actually. Those files are the memory contents at the time of the crash, and usually give a very good indication as to why it occured.

 

As for the upload, the Box upload widget uses our account, so you don't have to worry about storing the files. That's why we provide it.

 

As for the instructions in the link, you can ignore them for the most part. 

Link to comment
Share on other sites

  • 0

I tried uploading the dmp file to that wiki link, but it just hangs saying uploading. Here is the link to the memory dump (zip file), is this ok (it is about a 3GB file zipped)?

 

https://drive.google.com/file/d/0BxtYAJ7xBZGgbUZpR2RIcXZqLXM/view

From a quick look, the cause of the BSOD was due to memory corruption. 

 

I would highly recommend running a memory test on the system in question.

 

I've also flagged the dump for Alex, to take a closer look.

(https://stablebit.com/Admin/IssueAnalysis/13507)

Link to comment
Share on other sites

  • 0

This looks like a single bit error in the memory.

 

If you want details on the error, check out the above link, as Alex shows what the issue is exactly.
 
Specifically, a single bit was flipped. This could be just random (in theory cosmic rays could do this... again, in theory), but it is much more likely that you have a bad memory module in the system. 
Link to comment
Share on other sites

  • 0

Greatly appreciate this and makes sense since I have been dealing with this issue pretty much since day one of the build. I had run memory tests a few times but just the basic which is probably why it stayed under the radar. I am going to run the extended test but as well I am just going to buy some new RAM. Hopefully that solves the issue, I didn't want to have to rebuild my server and definitely didn't want to have to give up DrivePool as I built this server with Drivepool specifically in mind, I will post back with results once I get them

Link to comment
Share on other sites

  • 0

It may be worth reseating the sticks of RAM as well. 

In case dust or anything got into the socket, or if there is a loose connection.

 

And if you're using the built in memory diagnostics, and it comes up blank... then try memtest86+.

 

If that comes up blank... google your motherboard for memory issues. Also, make sure that your RAM is on the hardware vendor list. 

If that checks out "clean", then try running with just one stick of RAM and see if the issue comes back. If it doesn't add another. Until the issue returns. Then try using different slots on the motherboard. See if it's a bad slot.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...