Drive went bad - Now having backup problems

[larryf]

WHS2011, Drivepool and Scanner installed... Had an email about a bad disk and later found the server was hung. Tried rebooting but it froze again. On the 3rd reboot I could see the BIOS complaining about a 1TB disk, I powered down and removed it. Server booted up, used Drivepool to remove the disk and ordered a new one from Newegg. Drivepool now shows 100%, (but it took a while to get there).

 

It appears at this point my backup database is corrupted...  Nothing backs up overnight, I can't initiate a backup and one client shows backup status as unsuccessful. Showing the properties of any client gets the message "error retrieving backup data".  The Alert Viewer in the Dashboard shows an error, "A client computer backup has errors that must be resolved. before you can repair it, you must resolve all storage server errors"

 

What I've tried so far:

Tried repairing the backup database, got an 'unknown' error after about 15 minutes

Tried re-indexing drivepool, ran for over 24 hrs, still shows 100%

Tried removing commit.dat and rebooting, the file was recreated but still can't repair database  

Checked all of the Windows Server services, all are running

The replacement disk arrived, I installed it and added to the pool but still no joy here

 

Any suggestions on how to proceed from here? Should I run CHKDSK /R on the drivepool drive? 

 

Thanks in advance, 

 

Larry

[Christopher (Drashna)]

Unfortunately, it sounds like the Client Computer Backup database lost one or more files.

You may want to delete the entire contents of the folder and start from scratch. This may be the only way to proceed.

(you could just move the contents to a different directory, it produces the same result).

 

Also, if you grab the logs from "C:\ProgramData\Microsoft\Windows Server\Logs", and check the Event Viewer ("eventvwr.msc") and check for errors related to the services.  These may give you a better idea of what is going on. However, they can be a PITA to read. If you want, email them to me at christopher@covecube.com and I'll take a look at them.

 

As for the chkdsk, you cannot run that on the DrivePool drive, as it's an emulated disk.

However, you can run it on all of the disks in the pool. Just note that it will unmount the disks, and produce a "missing disk" error until the scan completes.

[Umfriend]

Sry if I misunderstand but did you have _no_ duplication and still remove a HDD that was not evacuated? In that case I would not be surprised your client backup database is indeed missing files and will not recover. Simply delete everything from it and restart seems like the best idea.

[larryf]

Sry if I misunderstand but did you have _no_ duplication and still remove a HDD that was not evacuated? In that case I would not be surprised your client backup database is indeed missing files and will not recover. Simply delete everything from it and restart seems like the best idea.

I did have duplication enabled on the backups...   The disk died 'hard' and had to be removed, I could not boot with it installed. 

 

I'm currently running Scanner on all the disks to see if it picks up anything, but I think you guys are right....   Move everything out of the directory (or delete) and start over.

 

thanks,

Larry

[Umfriend]

Ah, in that case I am worried. I have duplication enabled as well, also to ensure that I would not lose client backups should one HDD fail.

[larryf]

Ah, in that case I am worried. I have duplication enabled as well, also to ensure that I would not lose client backups should one HDD fail.

Yes, I'm a bit concerned over my backups too... If you Google "whs2011 backup database corrupt" you'll see a lot of others have seen the problem.

 

I was able to recover by copying the backup data to another folder, restart the backup service and back up all of the clients. I'll delete the old backups after a few days to get some space back. 

 

I don't think this was a Drivepool problem, I'm guessing things were corrupted when the drive died in the middle of a backup. I also think I aggravated the problem by trying to reboot with a  bad drive, I should have removed it when Drivepool complained about it.

 

Larry

[Umfriend]

Luckilly I also back them up as part of my Server Backup. And I'll accept that it i snot DP but drive-failure-induced-data-corruption.

[Christopher (Drashna)]

yeah, backup database corruption is/was fairly common. There have been updates to fix the issue. But if the database got corrupted by the failing disk (which is very likely), or the database wasn't duplicated... then this will definitely happen.

 

And I highly recommend checking the duplication status on the backup database folder. Make sure it's set to AT LEAST 2x. 

2x (or just 'duplication enabled") means that there will be a copy of any given file on two different disks. Setting it higher means it will be on more disks (and take up more space in the pool). Setting it to 3x may be a good idea if you suspect issues.

 

Alternatively, you could create a script to stop the backup service, copy the files and start it back up. You could then setup the Task Scheduler to run the script once a month/week/etc. That way, you would always have a good copy somewhere else.

[RobbieH]

Every time I lose a disk (which is suddenly quite often due to poor disk choices) I get corrupted backups. Usually I can fix them by running a repair, but over the weekend I had to remove my daughters laptop completely, run a repair, then add it back - all the way to the point that I had to remove the WHS Connector software from her laptop then reinstall.