Jump to content
Covecube Inc.
  • 0
Nissan_SR20_Man

covefs.sys using high CPU, causing system.exe (nt kernel & system) high cpu???

Question

All,

 

QUESTION...

1. Why is my drivepool (covefs.sys) chewing up my CPU??

 

BACKGROUND...

1. When copying large files, i can achieve 150+ MB/s R/W transfer speed easily.  Transferring a large *.MKV results in almost no PC CPU usage, at pretty fast transfers.  This is the ideal situation!

2. When doing other mundane task covefs.sys inside system.exe (nt kernel) is using 8 threads (normal) at very moderate to high cpu utilization.

3. Very high cpu usage is noted during plex transcodes, plex folder metadata refresh/updates, etc...  i understand transcoding is CPU intensive, i get that.  My issue is covefs.sys is using more cpu than plex transcoding process itself which is totally wrong.

 

PC SYSTEM...

1. AMD 940 Phenom II, 3.0GHz - stock clock/no overclock

2. Asus M3A32-MVP Deluxe MB

3. IBM M1115 (LSI (9240-8i) SAS2/SATA3 raid card.  Configured for pass through, no card raid.  All drivepool drives are controlled from this card

4. 12.7TB pool, using 8 drives.  3 different model types, SATA 2 and 3 mixed

5. WIN 7 Ultimate X64 - 100% VIRGIN INSTALL TODAY CHANGED NOTHING!!!  Only have Plex + Drivepool installed on a fresh bare metal install from MS DVD.  I recently decided to wipe clean, start fresh just to prove it was not me or my software

6. PC is media server.  It runs high performance power 100% of the time.  Runs Plex + Media Center Master + Utorrent 24/7.  That is it.  It is main repository for house + serves media to remote plex clients.  I use no power save crap, no cool & quiet, no sleep, no anything.  I need raw power, not power saving.

7. I have disabled indexing in the pool drive WIN7 properties window, no change

8. I have disabled indexing + search services from WIN7 thinking i was getting bamboozled my MS, no change

 

NOTES...

1. I know the hardware is old, give me a break please.  I can stream 4 1080p transcodes at the same time without drivepool and barely break a sweat.  With drivepool i can do 1, maybe 2.

2. It appears for some reason (my thought process) that during heavy transfer the Raid card kicks in and takes the info directly from source to destination.  During slower, multiple reads it uses covefs.sys and offloads the work the PC CPU???  is this possible???

3. Due to all pool drives residing on my PCIE Raid card, could the motherboard be gagging on the throughput on the PCIE bus or northbridge/southbridge controllers?

4. It appears to me that during low speed, multiple read/writes accessing lots of small files, it uses 10x more cpu than transferring large GB files.  It is literally night vs day difference between little files and big

5. I have confirmed everything mentioned above numerous times over two weeks and have used process explorer to pinpoint the covefs.sys usage.  The CPU usage  is even across all 8 threads.

6. If this cannot be resolved, i have two options...  Dump drivepool and go to Server 2012E OR Spend $700 for new MB/CPU/RAM to possibly prove this is not resolved.  Both of these possible resolutions cost me more money and heartache migrating to MS storage spaces.

 

Any help, guidance or opinions would be great.  I am frustrated due to never noticing this until lately and it is grinding my gears and burning up my CPU.  The PC is becoming useless during these times and is killing my performance elsewhere in the system.  Overall I can only transcode at roughly 30-40% due to covefs.sys using the other 50-60%

Share this post


Link to post
Share on other sites

11 answers to this question

Recommended Posts

  • 0

First, do you have "Network IO Boost" enabled in DrivePool? If you do, that could potentially cause that behavior. Though the option is disabled by default (for just this reason, actually).

 

Is this the release version (2.0.0.420) or is this a beta build (2.1)?

 

Do you have any antivirus software installed? Do you have any sort of disk tools installed (such as Acronis)? What about software from ASUS for the board? 

Any of these can absolutely cause issues.

 

As for the hardware... you should be fine. Actually, I have a M1115 cross-flashed to "IT Mode" (aka no RAID firmware at all, just an HBA card) in my system. I haven't had any issues with it, but it .... could. A way you could test this is to use StableBit Scanner. On the disks, there is a "Burst Test" option. This can be useful for diagnosing disk or controller issues. 

 

From the looks of it, you're running Plex, Media Center Master, and uTorrent, in addition to StableBit DrivePool. And these are the only things installed? 

If so, could you see able disabling them temporarily. See if it's not some interaction with one of them?

Additionally, do you have any of the drivers installed for the hardware? Especially LAN and storage drivers?

And what mode is the onboard storage controller using (IDE, RAID or AHCI)?

 

If that doesn't help, could you enable file system logging and try to duplicate the issue (most likely won't catch the issue here, but just in case).

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

After doing that, could you see about forcing a crash dump?

http://wiki.covecube.com/StableBit_DrivePool_System_Freeze

Share this post


Link to post
Share on other sites
  • 0

Drashna,

 

No network boost, as all media is local to machine.  Please read below as it is getting interesting on my end...

1. Plex App data + Plex Transcode temp data + Utorrent both are writing to the pool

2. I have started thinking about this as all of the above result in crazy read/write activity on the pool

3. I started offloading these program's directories off the pool to conventional HDD folders, essentially unloading the covefs.sys driver from the kernel, with amazing results.

4. Utorrent was first.  After moving this off of the pool, i saw a small noticeable difference.  Once again, good old process explorer+resource monitor (WIN7) to determine cpu loading while transcoding movies.  I ran multiple test with same movies with external clients to have similar loads on the pc

5. Plex transcode temp directory (location where transcodes go) was next.  This was a massive improvement due to how plex adds so many write threads.

6. Plex app data is in process (30GB) to be moved next

7. Moral of the story, with the pool holding media, and previously being read and written back during transcoding it was killing your covefs.sys due to being thread limited to 8 i believe.  Any ideas of data to back this up?

8.  Currently i have seen a 15X reduction in CPU usage by offloading the first two items.  I hope to see a bit more after the last.  But i still feel that drivepool should be able to cope.  It is silly of me to have everything R and W to/from the pool

9. Yes i have anti virus and i have added ignore rules to essentially rule it out.  it is avast for reference

10. from my viewpoint, i am essentially shoving an elephant or R/W data through the covefs,sys driver and holding off the cpu.  it loads up the queue and the cpu is very busy.  millisecond times are through the roof previously, now are essentially zero.  
I went from 100% CPU at 100% of the time to 8% all the time and small 100% peaks lasting 1-2 seconds.

Share this post


Link to post
Share on other sites
  • 0

From a technical point of view:

  • For reads and writes, CoveFS just forwards the I/O down to NTFS and the request continues down the driver stack as normal. There should be close to zero overhead.
  • The only long running background task that CoveFS performs is a "measuring" pass. It will enumerate all the files on the pool and compute size statistics. When this is occurring, you will see that in the UI under the "Pool Organization" bar as "Measuring...".
  • The most expensive operation that CoveFS performs, in terms of CPU cycles, is a file open / create. So theoretically, if you had a few processes enumerating and opening every file on the pool then that could lead to some significant CPU load (just enumerating files without opening each one is fast).

I used to run StableBit DrivePool 1.X on an Atom D525 (1.8 Ghz) and it was very usable.

Share this post


Link to post
Share on other sites
  • 0

So by forwarding the I/O, it is a small hit for the forward, but is tiny?  So, is this statement correct...If using crazy R/W's internal to the pool itself from 3 windows programs, the work overhead is small and logical R/W, not actual "true" R/W to physical disk?  The reason for asking is due to the highest cpu process during these time is the system "NT Kernel".  At 75-80% and the actual transcode process at 20-25%.  And covefs.sys is using 8 threads, consuming 70-75% of the system NT Kernel process.  It seems like I am burning up cpu for a read from a pool folder to a different pool folder write, neither of which is duplicated.  With folders out of the pool, all is better.  I simply do not get it.  There are crazy reads / writes at the same time.  It is like 50 and 50 for each program, so a big number to/from the pool itself.

 

Transcoding from a pool folder as a source to a pool folder as transcoded files is noticebly higher by a large factor vs source from pool and written to a non-pool folder on the same physical drive.  It is like i bottleneck somehow.  It is not measuring while this is high cpu noticed.

 

I need to quit typing and get logs and data.

Share this post


Link to post
Share on other sites
  • 0

9. Yes i have anti virus and i have added ignore rules to essentially rule it out.  it is avast for reference

 

We've had a number of people with issues from Avast lately. Interfering with both StableBit DrivePool (understandable as the AV uses a file system filter which can cause issues, and is why I asked), and with StableBit Scanner (which it SHOULD. NOT. EVER. cause issues with...)

 

Please uninstall Avast, and see if the behavior persists. If it does not,.... bingo. And looking into this interaction is on our to-do list.

However, if uninstalling Avast (not just disabling it... as that does not remove the said file system filters) doesn't help, then let us know. And please get the logs and dump to us...

 

 

And as for "overhead", as Alex has said, this should be minimal. I was using DrivePool on a Core 2 Duo E5200 with 4GBs of RAM (a HP MediaSmart Server EX495, actually), transcoding, downloading to, reading from, etc without any notifiable impact on the system (even with Network IO Boost, which uses more overhead), and minimal resource usage, except for when actively busy transcoding)

Share this post


Link to post
Share on other sites
  • 0

Interesting about Avast.  I have been using drivepool + avast for some time.  I just now noticed the cpu issue, obviously possibly related to an update.  Seeing how i have never noticed this before, i am either crazy, or avast got updated.  I will try and move the folders back in the pool and test.

 

I have found this on the web..."To clarify, include the DrivePool virtual drive but exclude the poolpart folders on the physical drives (if a physical drive is exclusively used by drivepool, you can simply exclude the entire physical drive)." 

 

This comment is about disabling file system scanning for the actual poolpart folders but include the virtual drive.  I found this very interesting as there is a documented link between scanning the poolpart folders.  Not sure if it helps.  I only disabled the virtual drive folder, never the poolpart folders.

 

For reference, i plan to baseline with folders outside, place them inside and disable file scanning as mentioned above, then after removal of avast in order to be objective across the board.  It should be really interesting.  Despite my frustration, this issue has taught myself a bunch.  As an engineer, diving into the weeds to find such problems is somehow satisfying, despite pulling my hair out...  If it was easy, everybody would be doing it...

Share this post


Link to post
Share on other sites
  • 0

Well, as both Windows and Avast both do automatically update by default... And some of the windows updates do affect the kernel... It's entirely possible that is just a "perfect storm" of issues here. Or maybe, you just hadn't noticed it before. It's hard to know for certain.

 

As for the exclusion, the "PoolPart.xxxx" folders are where the actual files reside, and where we pull all the information from. 

I haven't used Avast in a while, but in theory, you *should* be able to exclude the folders from scanning. But if the disks aren't being used for anything else... then excluding the disks may be helpful.

Specifically, when you read, write, list, etc files and folders from the pool, we're checking the contents of these folder. That's why there is the additional overhead, and why it should be VERY little overhead.

 

And any issue you can walk away from with more knowledge is, well a good thing. And I definitely agree with the "satisfying" part. :)

But I can definitely sympathize with the "frustrating" part, as well. Sometimes those really weird issues can drive you crazy.  So many moving parts make it a lot more complicated to troubleshoot, sometimes.

 

Oh, and speaking of which, what mode is the drive controller in? And are you using the Microsoft generic driver or the manufacturer's specific drivers for it?
I ask, because we've recently had somebody that had high CPU usage because of the drive controller's driver. It may be a shot at the dark, but I figured it was definitely worth mentioning.

Share this post


Link to post
Share on other sites
  • 0

Just wanted to pass along the info...  initial results are astonishing to say the least.  Avast appears to be the guilty party.  All antivirus real-time scanners should take note and should have exceptions "in theory" to eliminate the duplicated scanning or simply just the performance hit.  See info below.

 

1. In latest version Avast free

2. open avast console, navigate to antivirus tab on left, click it

3. inside the antivirus tab, scroll windows down to "EXCLUSIONS"

4. users need to open, copy and paste every physical drive hidden pool part folder path and paste them ALL in the exclusion list.  For every physical drive, there must be an exact pool folder path.  Easiest way is to open each in windows explorer and copy/paste from the address bar, just click the bar and it changes to the path.

5. DO NOT enter the pool drive letter folders, only the pool part.

 

For users concerned about disabling antivirus features, you can test this easily by generating a small file in notepad and saving it.  If your stuff works in real time, it should not even let you save it.  See image below of myself saving a file to the pool.  I also tested with the C: (system drive) and have identical results.  Test it yourself, just Google it.

 

Test notepad file being caught by avast after changing avast settings above

New_Bitmap_Image_5.jpg

 

Drivepool performance afterwards...  417MB/s total drive activity (previous was 180-200) - 209MB drivepool write to single un-duplicated folder.

 

New_Bitmap_Image_3.jpg

 

New_Bitmap_Image_4.jpg

 

 

I have copied my uTorrent temporary, inbound, torrent folders back to pool and restarted the program.  It is not causing any raise in cpu usage at all now.  I see pool speeds never before seen.

 

Plex media server 1080p transcoding speed as noted in the plex media server debug logs has jumped 8-9 times higher than previous values.  Speed of 1.0 = real time transcoding, i used to see 1-2.0 when avast was not bypassed.  Now 9-10 speed noted during utorrent traffic, plex scanner and media center master traffic consuming some CPU while transcoding.   :D

 

I have never seen any drivepool activity this high, let alone the entire system throughput on my LSI card.  Since i have all pool drives on the 8 SATA 3 inputs of the card, I think it is pretty darn good despite my MB being SATA 2, DDR2 on a V2.0 PCIE slot.  I actually deleted the avast scanning exceptions then added again just to confirm.  Night / day difference.

 

I will take a 800% CPU transcode performance increase for 5 minutes of tweaking avast settings.  And i wanted to drop big bucks for my CPU/MB...  my 2008 AMD 940 is going no where since this discovery.  I am speechless.  I still want to do more probing with avast to verify my finds and to ensure real antivirus scanning.  In theory if the virtual pool drive letter is scanned, it should be passed through to the physical drive and still be covered by antivirus protection??? despite adding the poolpart folder to be ignored???

Share this post


Link to post
Share on other sites
  • 0

Well, I'm glad I caught that bit with Avast then! 

I had initially, almost missed it. Good thing I didn't!

 

And I'm glad to hear that not only has this fixed the CPU and memory usage but greatly "boosted" the disk performance as well!

Great news.

 

And yes, Avast (or any other scanner) should detect the files based on the Pool access just fine.  The virtual "disk" acts like a normal disk, so there should not be any different in behavior there.

Share this post


Link to post
Share on other sites
  • 0

I wonder how many others using antivirus with real time scanning this could impact?  all scanners are not equal, however by default all scanners would read the pool folders + the pool drive.  If you need any more avast test let me know.  It appears to have resolved all of my problems.  Yes, despite covefs.sys being the consumption of the cpu, avast was simply working it to death.

 

Thanks again for all of the help and pointers, you only saved me about $1000 for a server refresh.  Well worth the money i paid buying the drivepool+scanner.

Share this post


Link to post
Share on other sites
  • 0

That really depends on the quality of the virus scanner, and how it's designed. Avast has had issues as long as I can remember. It was neat to have on WHSv1... but that definitely cause issues...

 

And no, we don't need any tests with Avast. the problem should be easily reproduce-able, and the solution definitely is.

 

And you are very welcome. And I'm glad that you didn't have to buy new parts, especially if you installed avast again, and likely experience the same issue!

And glad to hear it was worth buy our products! :)

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...