Jump to content
Covecube Inc.
  • 0

Reparse.covefs.* files


Question

About a month ago my Win10 system started freezing. I found that if I ejected some of my 14 drive pool disks, the system would unfreeze and behave normally. Same with bootup. Disks in, boot up would stall when it got to the desktop. Remove some drives and it would resume fine.

I checked SMART, ran chkdsk, swapped STATA cables, replace my SATA port multiplier cards (2) with a LSI SCSI card, formatted and reinstalled windows. Every time I thought I was making progress I would end right back to the crawling/freezing (sometimes BSOD) system.

I even replaced 2 drives that I thought were culprits as they seemed to always be ones I would have to remove. The problem never really went away. Until last night, I think. 

I found that on about 4 of my drives, including the two I had swapped out, my reparse\.covefs folder had 300k+ files in it, all created about a month ago on 2/5. These disks were causing the OS to stall, especially when DrivePool started measuring.

I put each offending disk in a USB enclosure, dropped into the command line, and ran del *.* in that folder. It took hours to clear out. However as the file count dropped below 100k, the system started running smoother. Eventually I cleared the folder each disk and (knock on wood), I think it worked. As of this morning DrivePool is measuring my pool, hopefully for the last time for awhile. When I get home we'll see if I'm back in business. I even think the drives I pulled out might still be good.

Any ideas as to what is going on?

Link to post
Share on other sites

14 answers to this question

Recommended Posts

  • 0
17 minutes ago, Rob Platt said:

About a month ago my Win10 system started freezing. I found that if I ejected some of my 14 drive pool disks, the system would unfreeze and behave normally. Same with bootup. Disks in, boot up would stall when it got to the desktop. Remove some drives and it would resume fine.

I checked SMART, ran chkdsk, swapped STATA cables, replace my SATA port multiplier cards (2) with a LSI SCSI card, formatted and reinstalled windows. Every time I thought I was making progress I would end right back to the crawling/freezing (sometimes BSOD) system.

I even replaced 2 drives that I thought were culprits as they seemed to always be ones I would have to remove. The problem never really went away. Until last night, I think. 

I found that on about 4 of my drives, including the two I had swapped out, my reparse\.covefs folder had 300k+ files in it, all created about a month ago on 2/5. These disks were causing the OS to stall, especially when DrivePool started measuring.

I put each offending disk in a USB enclosure, dropped into the command line, and ran del *.* in that folder. It took hours to clear out. However as the file count dropped below 100k, the system started running smoother. Eventually I cleared the folder each disk and (knock on wood), I think it worked. As of this morning DrivePool is measuring my pool, hopefully for the last time for awhile. When I get home we'll see if I'm back in business. I even think the drives I pulled out might still be good.

Any ideas as to what is going on?

After further research I may have my own answer to this. I think it was Plex's fault. I had read that Plex uses hardlinks, so while I wasn't accessing it via my main DrivePool drive, I was accessing it directly from one disk (yes it was still inside the PoolPart folder). I was letting the balancers run to copy the meta data to other disks. Perhaps my work-around really didn't solve anything. It did seem make the Plex WebUI work better. My guess is that is the reason for the high amount of reparse points.

I've since moved my Plex meta-data off onto a single SSD. Maybe I can leave the Database folder in DrivePool for the added redundancy. The meta-data can always be rebuilt.

If this sounds correct then maybe this will help someone else down the road.

Link to post
Share on other sites
  • 0

Good catch, and yeah, that could definitely cause issues. 

And yeah, the Pool doesn't support hardlinks, either.  unfortunately, plex's database/cache likes to use them heavily.  If you want to use them, then you'd need to change the folder location for that stuff to be outside of the pool. 

Link to post
Share on other sites
  • 0

So after trying to tackle this on my own. I'm at a loss. My pool is still causing huge latency and read failures. The service log is showing  Event ItemChanged took too long. 

I'm on version 2.2.5.1237. I just ran the Troubleshooter. The data was submitted successfully.

Plex's meta-data is longer in the pool drive.

I'm at a loss as to what could possibly be going on at this point.

Win 10x64 Home, fresh install. AMD Ryzan 7 1700x, 64GB ram, C: is on an nvme.

Re-ran chkdsk on all drives from the command prompt. All completed quickly and without error.

Switching to my drivepool drive and typing dir will cause the command window to hang. (edit: eventually the directory listing did pop up. took a couple of minutes)

Link to post
Share on other sites
  • 0

Hmm. With a really large number of files, the MFT might have become fragmented - if it gets truly bad enough, you can get effects like taking ages to list directories, access files and so on. Do you have a defragmenter than works on the MFT? UltraDefrag is free and can do so (highlight the physical - not virtual - disks to defrag, click Action menu then Optimize MFT).

 

Link to post
Share on other sites
  • 0
22 minutes ago, Shane said:

Hmm. With a really large number of files, the MFT might have become fragmented - if it gets truly bad enough, you can get effects like taking ages to list directories, access files and so on. Do you have a defragmenter than works on the MFT? UltraDefrag is free and can do so (highlight the physical - not virtual - disks to defrag, click Action menu then Optimize MFT).

 

Totally forgot about that. Running now against all physical drives. Might take the night. I'll report back when it's complete. Thanks for the reminder.

Link to post
Share on other sites
  • 0

Status update: I'm down to the last few disks out of my 14. It's probably going to take half the day to finish this optimization.

I have see no performance changes. 

Running the burst test in Scanner, on any of my disks, shows 0 B/s.

Once the MFT optimization is complete I'll reboot.

Last night I ran Troubleshooter. However I realized that file system logging wasn't enabled. So I enabled that last night - but haven't re-ran Troubleshooter yet.

DrivePool has been saying "Checking...." since last night.

Link to post
Share on other sites
  • 0

Are you running third-party antivirus (e.g. norton/avast/whatever), and does it make any difference if you uninstall it (not just set it to whatever it claims is "off", but actually uninstalled)?

Have you run a memtest? (yeah I know very unlikely to be the problem, but hey)

In addition to Scanner's burst testing, do you have any HDD benchmarking software? If you uninstall drivepool, is there any difference? When you disconnect enough disks that it starts working properly, does it matter which disks are disconnected?

 

Link to post
Share on other sites
  • 0
8 hours ago, Shane said:

Are you running third-party antivirus (e.g. norton/avast/whatever), and does it make any difference if you uninstall it (not just set it to whatever it claims is "off", but actually uninstalled)?

Have you run a memtest? (yeah I know very unlikely to be the problem, but hey)

In addition to Scanner's burst testing, do you have any HDD benchmarking software? If you uninstall drivepool, is there any difference? When you disconnect enough disks that it starts working properly, does it matter which disks are disconnected?

 

Right now I'm running a fresh install of windows 10. The only third party software I run is PrimoCache and it hasn't made any differences whether that was installed or not.

I turned the built in A/V off.

Initially it was running fine as long as the drives with the reparse files were not in. Once I had that mess cleaned up, it was still running fine until DrivePool started scanning/measuring. Then it would fall on it's face again. It seemed like every time I confirmed my disks were all fine, and said "this is it, I can finally blanace", DrivePool would die - which would cause the OS to stop responding (as anytime my virtual Disk was being enumerated there would be significant lag).

Last night when I posted my update, it would take nearly 2-3 minutes to query the DrivePool drive, then another 2-3 to try to continue querying it further. So for he hell of it, I told UltaDefrag to go ahead and do a full disk optimization on all of my HDD's, then I went to bed.

I woke up today to a balanced pool and a fully responsive DrivePool. I'm still only about 50% through the UltraDefrag.

I'm able to run Plex, browse my drive, browse my Media share from my Unraid box. Everything "appears" to be back to normal.

I've never seen this behavior from DrivePool before. I'm not confidant this is resolved. It would be interesting to see what the Troubleshooter logs say once someone has a chance to review them.

Edit: Scanner Burst Test still isn't working yet. Says it's running successfully but there's no I/O rate.

Link to post
Share on other sites
  • 0

I'm on a release build, but I noticed it's quite out of date compared to the beta's that have been released. Are there any I should consider updating to? That's one thing I haven't tried.

Also, just tried renaming a folder which has caused Explorer to hang and any further enumeration of my DrivePool drive to hang.

Link to post
Share on other sites
  • 0

I'm on the latest stable release (2.2.5.1237) for my backup machine and the latest beta release (2.3.0.1244) for my main server; haven't encountered any problems with either.

Link to post
Share on other sites
  • 0

Update -

I upgraded to Beta, no changes. I looked for more reparse files, found them and deleted them. Perhaps a subtle difference. Hard to say. Wish I knew why they were being created.

However, while I was waiting for the DrivePool to unfreeze, I started working directly on my disks - which, themselves were super responsive. 

Remember when I said I renamed a folder and it froze? It happened to be my Dropbox folder (Dropbox wasn't even installed at this point, this was left-over from the previous OS), which I was going to redownload anyway. I went through each disk to to delete the folder. Once the folder was deleted I noticed that DrivePool was back to behaving normally.

So far, it's been perfectly fine since my last post.

I'll update this thread after I've done some more testing.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...