Jump to content
  • 0

Recovery from File System Damage affecting multiple drives


keinreis

Question

Note to moderator - If this question would be better answered in the Scanner forum, please feel free to move it. 

 

Admittedly, I'm more of a "casual user".   Ignorance was bliss in my case... until it wasn't. 

I had been on an older version of DrivePool for a long time by the time I finally got the memo, and installed the update to 2.20.906 - which was around the beginning of the April.  Almost immediately afterwards, I started having file access issues on some of the drives in my pool.  I was unable to move files from certain folders on certain drives in File Explorer.  That prompted me to open Scanner for the first time in a long time to see if there was something wrong with those drives.     

At that point, I noticed that Scanner was stuck at 57% on one of my drives, and that it had never completed a full surface scan on that drive, which had been in my system for months.   

I contacted support, and Christopher recommended I upgrade to the latest version of Scanner, which I did.  I also ran the Troubleshooter on both DrivePool and Scanner, and uploaded the log files.  He didn't see anything that stood out in my logs, but he did point out that having both MalwareBytes and Acronis installed could be the cause of my problems.  To quote him, "These use file system filters that intercept and can modify all file access. It could be that one, both or the combination of these filters is causing this issue."  He recommended uninstalling one of both of them, to see if the file access problem went away.  Has anyone else had any issues with these two programs?  I've come to rely on both of them, and don't want to get rid of them unless it is really necessary. 

Well, before I could even try Christopher's recommendation of uninstalling MB or Acronis, the newer version of Scanner I'd installed started alerting me to all sorts of issues on my drives.  Something must have really nailed me, because pretty much all five internal drives in my pool, and my OS drive, plus three 8TB Seagate External USB drives ended up with File System Damage (errors in the MFT bitmap, or MFT mirror corrupted).  I ran CHKDSK /R on all the drives, but it didn't fix the MFT issues.  A few of the drives, including my OS drive, also had damaged sectors - just one sector per drive.  I was able to recover the files that were affected.  All my data was backed up to the 3 8TB Seagate Externals, but two of those also had MFT Bitmap errors, and the third one had a bad sector.  So I knew I had to work out a recovery plan.  That's where I could use some help.   

I purchased a new SSD for the OS drive, and one new Seagate 8TB Ironwolf HDD for my pool.  I also purchased three new Seagate 8TB External drives for backup.  

I am now in the process of moving data off the drives with issues, and on to new drives.  I will also be purchasing a couple more 8TB drives as soon as I can to get enough capacity to start using duplication - which I have never used before. 

So far, I have removed one of the old HDD's from the pool, and installed the new 8TB Seagate Ironwolf HDD, which has been added to the pool.  Installing the new 8TB Drive has given me enough free space to remove another one of my old internal HDD's which has MFT bitmap errors.  That drive which will be removed next only has MFT bitmap errors, and no bad sectors.  It's only been in service for few months.  I'd like to reuse it, if possible.  Would simply reformatting that drive resolve the MFT bitmap error issues?   

If reformatting the drive will work, then I'll probably do the same thing with the rest of my internal and external drives which only have MFT bitmap errors.  Again, those drives aren't throwing any SMART errors.  They don't appear to have any physical damage/bad sectors. 

Ideally, then I would be in a position to re-purpose some of the old Seagate 8TB external drives for local backup again.  I would have a total of 5 of them, which would match the capacity of my internal drives in my pool.  Dealing with that many external USB drives is a bit of a pain, though.  I was thinking of shucking the external drives and putting them in some sort of external enclosure.  Are there any good external enclosures on the market? 

Thanks for the help.  
 

 

Edited by keinreis
Changing title
Link to comment
Share on other sites

10 answers to this question

Recommended Posts

  • 0

Update 4/29:  I thought I was out of the woods and could move on with the recovery process, but it appears something is corrupting the file system on nearly every drive on my PC.  This morning, upon booting up, I was greeted with whole new round of File System Damage warnings in Scanner.  At this point all five of the drives in my pool have File System Damage warnings, including the brand new 8TB Seagate Ironwolf drive I just installed yesterday.  I ran CHKDSK /F on all five of the drives, and only one reported any issues that needed to be corrected - "MFT Mirror Corrupted".  This was supposedly fixed by CHKDSK /F.  The other four drives didn't report any file system issues at all.  

I don't think I can disregard these warnings as false alarms.  I did discover one folder that contains a large number of files is no longer accessible.  File Explorer won't shows any of the files in that folder, let alone allow me to open, or move them.  So far, I've only found one folder with that issue, but there could be others.

Hardware-wise, I should mention that I have made some changes recently - after the first round of File System Damage and Bad Sector warnings earlier this month.  I installed a brand-new, more powerful power supply, and all new SATA power and data cables.  I am certain the physical cable connections are solid, but I have not run any Burst tests to see if there are SATA controller issues.  Would that be a logical next step to take in the troubleshooting process?   

I also run a quality 1500-sized UPS which is not showing any "events", so I'm not apt to blame power issues for this. 

Link to comment
Share on other sites

  • 0

While I'm certainly not as good at diagnosing Drivepool/Scanner issues as the team here, things I would tend to suspect first:

  • The update to the Drivepool software, from very old to most recent.  Might have made changes to the COFS (file system) that the old schema simply wasn't 100% compatible with, and which started causing issues on pooled disks.
  • The controller(s) on the motherboard, since I don't think you have an add-in SAS/SATA card.  Possibly even cables/connections.
  • The Scanner software itself.  Christopher might have a better Beta version for you to use (he often does).
  • The filters imposed by both MalwareBytes and Acronis.  They might be interfering with disk operations, or colliding with the Performance feature in Drivepool for enabling/disabling filters.

Also - have you taken any of the drives and plugged them into any other workstations with comparable SATA levels (i.e. SATA3 controller+cable) to check them on a known good system?  I'd be particularly interested in the newest Ironwolf, and what tests on it would show.  You could do the same for the others, to isolate any potential problems from your current software/OS/hardware layers, and *just* test the drives, and possibly their cables.  If they were damaged and could be repaired, then placed back into the current system, they would act as tests themselves...  if they re-corrupted after off-system repair, you'd know it was something about your system and not any of the drives.

Link to comment
Share on other sites

  • 0

So I am having similar issues.  My pool just vanished and all 5 drives appear to be corrupted.  What I am doing now is running Stellar Pheonix on each drive and moving my files to a new set of drives.  its a huge PITA but I guess I should be happy I have my files.

BTW I have used Drivepool since the Beta and it has been virtually bulletproof for me until this issue.  Having said that I am moving off of my Windows Server 2012 R2 and on to a NAS instead.  WS2012 has been too much of a PITA and a NAS just works for a home network.

Link to comment
Share on other sites

  • 0
22 hours ago, Jaga said:

While I'm certainly not as good at diagnosing Drivepool/Scanner issues as the team here, things I would tend to suspect first:

  • The update to the Drivepool software, from very old to most recent.  Might have made changes to the COFS (file system) that the old schema simply wasn't 100% compatible with, and which started causing issues on pooled disks.
  • The controller(s) on the motherboard, since I don't think you have an add-in SAS/SATA card.  Possibly even cables/connections.
  • The Scanner software itself.  Christopher might have a better Beta version for you to use (he often does).
  • The filters imposed by both MalwareBytes and Acronis.  They might be interfering with disk operations, or colliding with the Performance feature in Drivepool for enabling/disabling filters.

Also - have you taken any of the drives and plugged them into any other workstations with comparable SATA levels (i.e. SATA3 controller+cable) to check them on a known good system?  I'd be particularly interested in the newest Ironwolf, and what tests on it would show.  You could do the same for the others, to isolate any potential problems from your current software/OS/hardware layers, and *just* test the drives, and possibly their cables.  If they were damaged and could be repaired, then placed back into the current system, they would act as tests themselves...  if they re-corrupted after off-system repair, you'd know it was something about your system and not any of the drives.

Thank you for the suggestions.  I plan to test the drives in another machine - starting with the brand new Ironwolf drive.  Unfortunately all my other desktop PC's are very old - having only SATA II.  I do still have my trusty HP EX495 MSS up and running.  This is not the server with File System Damage issues.  The HP doesn't even have DP on it, and there's an open drive bay.  Anyone remember if those have SATA III?

Link to comment
Share on other sites

  • 0

I think functionally (for a surface test) a SATA 2 connection would work fine.  If I was going to diagnose the drive fully, I'd prefer a straight connection rather than a bay.  But both might work.  Typical repair/test utilities might have better luck with a normal cabled connection.

Link to comment
Share on other sites

  • 0
On 4/28/2018 at 11:55 PM, keinreis said:

I had been on an older version of DrivePool for a long time by the time I finally got the memo, and installed the update to 2.20.906 - which was around the beginning of the April.  Almost immediately afterwards, I started having file access issues on some of the drives in my pool.  I was unable to move files from certain folders on certain drives in File Explorer.  That prompted me to open Scanner for the first time in a long time to see if there was something wrong with those drives.     

That's a known issue with the current release.  This is fixed in the RC build: http://dl.covecube.com/DrivePoolWindows/rc/download/StableBit.DrivePool_2.2.1.922_x64_RC.exe

On 4/28/2018 at 11:55 PM, keinreis said:

errors in the MFT bitmap, or MFT mirror corrupted

Unfortunately, these seem to be pretty common. Especially if the system has rebooted unexpectedly (BSOD, for instance). 

On 4/28/2018 at 11:55 PM, keinreis said:

Would simply reformatting that drive resolve the MFT bitmap error issues

Running "CHKDSK /F" should fix that, actually.  Don't run it with the /scan option, as that will skip this. 

On 4/29/2018 at 12:58 PM, keinreis said:

I don't think I can disregard these warnings as false alarms.  I did discover one folder that contains a large number of files is no longer accessible.  File Explorer won't shows any of the files in that folder, let alone allow me to open, or move them.  So far, I've only found one folder with that issue, but there could be others.

File system errors happen from time to time.  And if you had any BSODs or powered off the system without shutting it down, this can also happen. 

It's not a "show stopper", but it's not something to ignore either. 

Link to comment
Share on other sites

  • 0
On 5/1/2018 at 11:31 AM, Christopher (Drashna) said:

That's a known issue with the current release.  This is fixed in the RC build: http://dl.covecube.com/DrivePoolWindows/rc/download/StableBit.DrivePool_2.2.1.922_x64_RC.exe

2.2.1.922_x64_RC has been installed.  But I have not been noticing any file access issues lately.  My main issue now is the multitude of File System Damage warnings I am getting on every drive in my pool, even after repeated repairs via CHKDSK /F.  

On 5/1/2018 at 11:31 AM, Christopher (Drashna) said:

Unfortunately, these seem to be pretty common. Especially if the system has rebooted unexpectedly (BSOD, for instance). 

Running "CHKDSK /F" should fix that, actually.  Don't run it with the /scan option, as that will skip this. 

File system errors happen from time to time.  And if you had any BSODs or powered off the system without shutting it down, this can also happen. 

It's not a "show stopper", but it's not something to ignore either. 

I also took Jaga's advice and tested every one of these drives in another PC.  I ran CHKDSK /F on each drive.  No file system issues were found on any drive.  But just to be sure, I also used the drive manufacturer's diagnostic software to test each drive.  No SMART warnings, no file system errors, no bad sectors.

 I then reinstalled these drives back in my server, and reset the "File System Health" status to "unchecked".  The status changed Immediately to "File System Damaged" on each drive as soon as I did that.  I ran CHKDSK /S on each drive again, and it found MFT Mirror and bitmap errors on every drive in my pool.  They had been installed for less than 5 minutes.  Not enough time (even for me) to cause an unexpected shutdown or BSOD. 

Any additional suggestions would be appreciated.  

   

Link to comment
Share on other sites

  • 0
15 hours ago, keinreis said:

2.2.1.922_x64_RC has been installed.  But I have not been noticing any file access issues lately.  My main issue now is the multitude of File System Damage warnings I am getting on every drive in my pool, even after repeated repairs via CHKDSK /F.  

If this is for StableBit Scanner, then try installing this version: 

http://dl.covecube.com/ScannerWindows/beta/download/StableBit.Scanner_2.5.2.3162_BETA.exe

 

IIRC, there was an issue with false positives, and the newer version should fix this.

Link to comment
Share on other sites

  • 0
On 5/3/2018 at 11:41 AM, Christopher (Drashna) said:

If this is for StableBit Scanner, then try installing this version: 

http://dl.covecube.com/ScannerWindows/beta/download/StableBit.Scanner_2.5.2.3162_BETA.exe

 

IIRC, there was an issue with false positives, and the newer version should fix this.

Done.  Seems to have worked.  After installing that version of Scanner, I am no longer getting File System Damage warnings on all the drives in my pool.  

Though I'm not certain these were all "false positives".  Throughout my many rounds of testing the drives, CHKDSK /F  had reported what appeared to be legitimate MFT Mirror Corruption and/or Bitmap Errors.  But those have likely been resolved/repaired, and no new warnings have been triggered.  So I think (hope) I'm good.  

I am now in possession of 5 new Seagate 8TB External USB 3 drives.  My plan was to shuck those, and install the drives in some sort of external enclosure.  I prefer to avoid USB 3, and I've had plenty of issues with eSATA in the past.  My server mobo has USB 3.1 Gen 2 ports (10Gbps) on it (1 Type A and 1 Type C).  I see that there are a few enclosures with USB 3.1 Gen 2 connections.  The speed of USB 3.1 Gen 2 is appealing, but would it also be prone to the same kinds of disconnection issues as USB 3.0/3.1 Gen 1?

Next month, I will get two more 8TB internal (NAS) HDD's, which will finally allow me to turn on duplication on my entire pool.  This has been a process, but progress is being made.

Thanks for all the help!

 

Link to comment
Share on other sites

  • 0

Yeah, USB will still have some of the same issues (since the specs allow for them, actually).  eSATA is better, but ONLY if you get "good" chipsets. 

If you want good externally attached storage, you want to use SAS. But that gets expensive, FAST.  

Internal is king, though. Especially if you use a SAS controller. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...