Understanding my long backup time

[Breeze]

I just moved to WHS 2011 from WHSv1. The only thing I use WHS for is backing up PCs; I manage my assets on another machine. On WHS, I'm more concerned about data integrity and the ability to vacate and replace a failing drive, so I opted to install 2011 on my older WHSv1 hardware and to use Drivepool/Scanner. I installed these versions:

 

Drivepool 2.1.0.553 RC

Scanner 2.4.0.2929

 

I also used PerfectDisk on my WHSv1 machine, so I installed v13 on this build as well. It's set to use Optiwrite and to ignore the pooled disk. I used Optiwrite on WHSv1 where it had no significant impact on performance.

 

The system can typically maintain 50 MB/sec on the Gbit network it's on. But the backup of my first Workstation, with about 2.5 TB of data, has been running for 17 hours now and is almost 70% complete, which will put the total backup time at about 24hours. This is roughly double what it should be taking.

 

The system resources seem more than adequate for the task. I have noticed that sometimes there's network activity, but little disk activity. The rate the Drivepool applet shows varies between 10-12 MB/s and about the expected 50 MB/s. I currently have 4 - 1TB drives in the pool.

 

So I'm wondering where the bottleneck is. Is it the Drivepool RC? Or perhaps settings in Drivepool, Scanner, or PD, that I need to optimize and that I'm unfamiliar with? Thanks for any tips.

[danfer]

I am experiencing the same.

I am running a WHS 2011 machine with DrivePool 2.1.555 and Scanner 2.5.0.3006.

 

Usually I can move files with 100-120 MB/sec, now I am at 12 MB/sec 

[Christopher (Drashna)]

Is this for the Client Computer Backup? 

If so, the bottleneck is the backup service usually. In additional to transferring the data over the network... it's not saving it to disk or a "normal" format. It's saving information about the disk, the different sectors, running sector comparisons (for de-duplication), and updating 4GB chunks with non-duplicated data and it's "index" for that data. 

This isn't exactly a simple or straight forward process, so it will take longer "expected".

 

Also, the defragmentation filter *could* be interferring with disk access here, slowing it further.

I am experiencing the same.

I am running a WHS 2011 machine with DrivePool 2.1.555 and Scanner 2.5.0.3006.

 

Usually I can move files with 100-120 MB/sec, now I am at 12 MB/sec 

The only change is updating DrivePool? If so, what version where you using that was running much faster? And does reinstalling that version help?

 

Also, could you enable file system logging (gear icon -> troubleshooting -> "enable file system logging") and duplicate the copy. Then use the following link to upload the logs:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

[Breeze]

Thanks for the observations, Christopher. It's now past the 24 hour mark, with 16% left to go. I understand that this is a lot of data, and that there's a lot of management involved in doing client backups, but it's still considerably longer than it took with WHSv1. The only difference, apart with the root OS which I would expect to be at least as efficient, is DrivePool. Everything else, hardware, network & NICs, even drives  and clients, are exactly the same.

 

I think it may also have something to do with the drive balancing. I get the feeling that every so often, there's a time where the drives in pool are being balanced and then the backup resumes. Is this the case? Every so often the Performance part of the Addin reports "No disk activity on the pool" for a while... and then resumes.

 

I'm also wondering if DrivePool & Scanner consider drive speed at all. Thanks to Scanner, I do know that the drives are fine, but shouldn't each drive's read/write performance be considered and factored into the performance of the pool? Seems to me one slow drive would bring the speed of the entire pool down. Without testing I can't really tell, but I think one of the drives in the pool is consistently slower than the others; that could be the problem.

 

BTW, are you interested in my log?

[Breeze]

Done! It took 30 hours. I'm going to have to figure out if this can be improved. I'm hoping the incremental updates are going to be significantly faster.

[Christopher (Drashna)]

Well, even if the hardware is identical, the drivers are most certainly not. It could be an issue with a driver causing performance issues. Copying files to and from the pool (especially over the network) would be a good way to see if that's the issue.

 

There is a post of mine from the "We Got Served" website that goes over a lot of the common network performance issue stuff. It's meant for WHSv1, but most of then stuff still applies.

http://forum.wegotserved.com/index.php/topic/8335-before-you-post-media-stuttering-playback-issues-performance-irregularities/

 

 

As for performance.... Well, DrivePool absolutely factors in disk speed when it comes to read striping. And StableBit Scanner throttles it's scans based on disk and controller activity, to prevent bottlenecking.

But as for writing data... well, there are some factors used in determining placement, but there should be little to no noticeable impact on performance.

 

 

 

And yes, further backups should be incremental, and not take much time, depending on how much the data has moved around on the disk (remember, sector based backup)

[Breeze]

Well, even if the hardware is identical, the drivers are most certainly not. It could be an issue with a driver causing performance issues.

Actually, the driver angle is a valid one: my graphics card currently is using the default VGA driver and doesn't have a proper WS2008-R2 driver. There may well be other issues, though none glaringly obvious.

 

BTW, the first auto-backup worked last night, but a second client backup got hijacked by "update Tuesday"... and now I can't connect to the dashboard from that client... (%!?#&/$!!!). Teething pains.

 

Thanks for all the tips. I'll definitely take a closer look at the hardware config and do some benchmarking.

[Christopher (Drashna)]

Well, if you're not sleeping the system, the GFX driver shouldn't matter. Though... you never know.

But updating drivers is usually a good first step when troubleshooting issues or performance. As is firmware (motherboard, controller cards, etc). 

 

 

And I totally feel you on the Patch Tuesday. "Oh, I need to reboot to test this real quick." "Installing updates.... X out of 20". NOO! 

 

 

And as for that link, that's based on a couple years worth of troubleshooting performance issues. 

 

Oh, and something else that could affect performance here: Antivirus on the "server" machine. As most use a file system filter to intercept file access.... it can adversely affect DrivePool. usually excluding all the "PoolPart" folders helps though.

[Breeze]

I found something strange: when I checked up on the WHS machine, I noticed the DVDROM light was going on and off constantly. It turns out I had left the WHS 2011 disk in it. After I removed it, I checked the Windows System logs and found a huge list of consecutive error messages whose source was "cdrom": "The driver detected a controller error on \Device\CdRom0."

 

Not sure why this was happening; never seen this before. Could it be related to Scanner? FYI...

[danfer]

The only change is updating DrivePool? If so, what version where you using that was running much faster? And does reinstalling that version help?

 

Also, could you enable file system logging (gear icon -> troubleshooting -> "enable file system logging") and duplicate the copy. Then use the following link to upload the logs:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

I also updated my network drivers. After some time I found out, that the update changed my network settings from 1000mbit to 100mbit.

I am totally sorry for blaming you     

 

After changing this setting, I have no problems anymore.

Nevertheless thanks so much for your quick feedback and support on that 

[Christopher (Drashna)]

I found something strange: when I checked up on the WHS machine, I noticed the DVDROM light was going on and off constantly. It turns out I had left the WHS 2011 disk in it. After I removed it, I checked the Windows System logs and found a huge list of consecutive error messages whose source was "cdrom": "The driver detected a controller error on \Device\CdRom0."

 

Not sure why this was happening; never seen this before. Could it be related to Scanner? FYI...

It ... could be, but I'm not sure. In theory, Scanner shouldn't be pinging optical drives.

 

I also updated my network drivers. After some time I found out, that the update changed my network settings from 1000mbit to 100mbit.

I am totally sorry for blaming you     

 

After changing this setting, I have no problems anymore.

Nevertheless thanks so much for your quick feedback and support on that 

That would totally bottleneck access! (12.5MB/s max theoretical, vs 125MB/s for gigabit, so .... yeah)

And absolutely no problem. But I'm glad I could point you in the right direction there.